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REGRESSION TESTS FOR SEVERAL SAMPLES 


HAROLD GULLIKSEN 
AND 
S. 5. WILKS 
THE EDUCATIONAL TESTING SERVICE 
AND 
PRINCETON UNIVERSITY 


In many situations it is desirable or necessary to administer a 
set of tests to several different groups, and to ask if the results ob- 
tained in the different groups may be regarded as being essentially 
the same in some sense. In the case of two variables (one dependent 
and one independent) one may, for instance, ask if the errors of esti- 
mate and the regression lines may be regarded as being the same 
for the populations from which the different groups are drawn. 
For this case, the present article considers tests for three hypothe- 
ses regarding the populations from which the different groups are 
drawn: (a) H,, the hypothesis that all standard errors of estimate 


are equal; (b) H,, the hypothesis that all regression lines are par- 
allel, (assuming H,); and (c) H,, the hypothesis that the regres- 
sion lines are identical, (assuming H,). Test criteria for these three 


hypotheses and their sampling theory for large samples are pre- 
sented. The results are extended to the case of several independent 
variables. An illustrative problem is presented for two groups, two 
independent and one dependent variable. 


I. Introduction 

In many situations it is desirable or necessary to repeat an in- 
vestigation several times and to ask if the several repetitions have 
given essentially similar results. In polling, for instance, one may 
wish to know if the results obtained by Interviewer A are in agree- 
ment with those obtained by Interviewer B or if the results obtained 
in State A are essentially similar to those obtained in State B. In 
evaluating tests it is often desirable to know whether the results on 
validity obtained in College A are essentially similar to those ob- 
tained in College B. 

In the case of samples from bivariate populations, one may wish 
to know whether the samples may be regarded as having come from 
populations in which the means, variances, and covariances are the 
same. More precisely stated, if O, is a sample from a population with 
means uz, and wy, , standard deviations o,, and oy, , and correlation 


coefficient Poy. (k=1,2,--- ,K), we may ask whether 
91 
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He — Le 
My = by 
iad Oz 
Fy, — Fy 
Pru, — Pry 


fork=1,2,---,K. 

The solution te this problem for the case of random samples 
from bivariate normal populations has been given by Pearson and 
Wilks (1). However, it may happen that samples 


O.(k=1,2,---, K) 


are not random samples from bivariate populations; they may come 
from different sections of the same bivariate distributions where the 
drawing of the samples has been controlled with respect to the x 
variable. For example, as shown in Figure 1, it may be that, in sam- 
pling a bivariate distribution, sample O, comes from the upper 2/3 
of the distribution (indicated by vertical lines); sample O. from 
the lower 2/3 of the distribution (indicated by horizontal lines) ; and 
sample O; from the middle portion (indicated by both sets of lines). 
In such a situation the means of the different samples may differ 


7 











FIGURE 1 
Illustrating Explicit Selection on x and Incidental Selection on y 
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widely, and similarly for the standard deviations and the correla- 
tion between x and y, and yet one may be sampling from the same 
universe, each sample being from a different portion of it with re- 
spect to the values of x. 

If one wishes to test the hypothesis that his sampling has been 
done from different portions of the same universe where selection 
has been made with respect to x, then clearly a test of equality of 
means, variances, and covariances cannot be used. In a situation of 
this kind, one can say that selection has been made directly on the 
basis of one of the variables (say x) and that any selection which 
has occurred on the other variable has not been due to direct and ex- 
plicit selection on that variable but has been an incidental selection 
due to correlation of the second variable with the first. For exam- 
ple, in the diagram indicated in Figure 1, the selection has been on 
the basis of x; and selection on y has occurred only incidentally be- 
cause of the correlation between y and x. Another way of stating 
this is to say that errors in the y score have not affected the selection 
process, but errors in the x score have affected the selection process. 
In this sense, then, selection is based explicitly on x and only inci- 
dentally on y. In such a case it is possible to test the various sam- 
ples to see whether they can be regarded as having come from popu- 
lations with the same regression line. 

If selection has been directly on the variable labelled x and only 
incidentally on the variable labelled y, then it is appropriate to ask 
whether the samples could reasonably have come from populations 
(or different parts of the same population) in which the regression 
of y on x is the same for the various populations, (or for the various 
parts of the same population in case the samples are from different 
parts of the same population). It is clear that if one selects explicitly 
on x, the regression of x on y cannot possibly be the same for the 
various groups. However, when one does select explicitly on x from 
a bivariate distribution, this selection will not systematically affect 
the slope of the regression of y on # nor will it systematically affect 
the variation about this regression line (e.g., the error made in esti- 
mating y from 2). 

Thus, if one takes K samples from bivariate universes (or from 
a bivariate universe) and if these samples differ in mean, variance, 
and covariance, one may find that the standard error of estimate and 
the slope of the regression line remain relatively constant from sam- 
ple to sample despite variation in means, variances, and covariances 
of the K samples. 

In this paper we shall state and illustrate criteria for testing 
hypotheses regarding constancy of the standard error of estimate, con- 
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stancy of slopes of regression lines, and equality of intercepts of re- 
gression lines from sample to sample. The criteria are derivable by 
straightforward application of the Neyman-Pearson likelihood ratio 
test theory. Some of them are equivalent to analysis of covariance 
methods as applied to several samples involving regression variables. 
The reader will find the standard small-sample analysis of covariance 
treatment of some of the hypotheses considered in this paper in Chap- 
ters 12 and 13 of Snedecor’s Statistical Methods (2). The purpose 
ot this paper is to present large-sample tests for the hypotheses con- 
sidered from the point of view of Neyman-Pearson likelihood ratio 
test theory. 


II. Bivariate Case 
We shall first consider the tests for the bivariate case and then 
generalize to the case of several variables. 
We shall use the following notation: 


k= subscript designating the k-th sample 
K designates the number of samples 
(k —Je | pitty K ) 


4,j=subscripts designating the i-th or j-th individual in a 
specified sample 


m= number of individuals in the k-th sample 
(i=1,---, mm), (j=1,---, ™%) 


X;; designates the independent variable score for the i-th 
individual in the k-th sample 


Y;; designates the dependent variable score for the i-th in- 
dividual in the k-th sample 


O, designates the k-th sample and consists of the pairs of 
scores (Xin, Yia) , (Xie, Vie) ,+++ 5 (Xen, » Yin.) 


= Xxi 
xy. =— , the mean X-score for the k-th sample 
Nr 





> Yui 


i=1 





e 
I 


, the mean Y-score for the k-th sampie 
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es = Xz; —Xy. , the x deviation score for the i-th individual in 
the k-th sample 


Yui = Yui — Yy. , the y deviation score for the i-th individual in 
the k-th sample 


K 
N= ™ , the total number of individuals in all samples 
k=1 


KE ™ 


= Xi 
= _ i= 


v , the grand mean over all samples for x 


K 

= > Yui 
a k=1 i=1 
| —— the grand mean over all samples for y 


“ 


>I 


~ 


the deviation of the mean of the k-th sample 
from the grand mean for the independent vari- 
able 


2». = Xx. — 


y', = ¥,»—~ ¥. , the deviation of the mean of the k-th sample 
from the grand mean for the dependent vari- 
able. 


We shall consider O; as a sample of m pairs of scores (Xx: , Yxi); 
(4 =1,2,---,m™) drawn from a population in which Y;; is consid- 
ered to be normally distributed with mean o, + 6,X;; and with vari- 
ance o;?. This means that O, is considered as a sample from a bivari- 
ate population in which Y is normally distributed with standard er- 
ror of estimate o; around the regression line of y on x having the equa- 
tion Y =o; $f;.X , such that the values of X in the sample are “fixed,” 
i.e., regarde@Ms being selected in advance without error. The prob- 
ability elem@@& of Y;; (ie., that Y;; will fall between Y;; and Yx; + 
dY;;) is 


1 
fl ~ (YRi-0x-Be Xn i)? 
(2042 





a dY;;. (1) 
V 220% 
In drawing all of the samples O,, O., --- , Ox, the probabil- 
ity element for the Y;;(K =1,2,---,K;i=1,2,---,m) is 

nm Ko om, (Yxi-Qe-BeXni)? 

K ; kt “Si a. 

IT (=) € IT WdyY;;. (2) 

ba \220;? k= i=1 
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We wish to test three statistical hypotheses regarding the popu- 
lation parameters a, , 6, 0? (kK =1,2,--- , K) on the basis of the 
information supplied by the samples O,. Let the hypotheses be de- 
noted by H,, Hz, and H, and defined as follows: 


(a) 4H, is the hypothesis that the population variances (stand- 
ard errors of estimate) o,?, 02”, --: , ox? are all equal no 
matter what values the a, and f;, may have. 


(b) Hz, is the hypothesis that the slopes #,, 62, --- , Bx of the 
population regression lines are all equal assuming Hy, 
is true (i.e., that the variances are all equal), no mat- 
ter what values the regression intercepts a, , @2,-:-, Gx 
may have. 


(c) HA, is the hypothesis that the regression intercepts are 
equal assuming H; to be true, i.e., that the variances 
are ecual and the regression slopes are equal. 


To test these three hypotheses we need test criteria which can be 
computed from the sample O;. Appropriate criteria may be obtained 
by the Neyman-Pearson method of likelihood ratios. 

The likelihood ratio 4, for testing H, is defined in the following 
way: Let M, be the maximum of expression (2) with respect to the 
parameters a; , §,, 0.2 (k =1,2,---, K) for fixed values of the Xx; 
and Y;;. Let m, be the maximum of (2) with respect to o,, /;,, 0? 
(k=1,2,---, K) after putting o,? = o.2? —--- = ox? = o’, for fixed 
values of the X;; and Yx;; 44 is defined as the ratio ms4/M,. The 
test criterion which we shall use is G4 = —2 log. 44. We shall not 
present the details of obtaining m, and M,, but shall merely state 
the result. (For details of the application of the likelihood ratio meth- 
od to problems similar to this one, see pp. 157-175 in reference 3.) 
The expression for G, is 


1 * x 1 
G.=Nlos. | 72S | —Smlog,| = s, | ; (3) 
N Nt, 


LV k=1 k=1 


where 


D Lei Yi 


i=1 


nk 
S.=3 x? — ———_, (4) 
as p> xi” 
=1 


the notation being defined at the beginning of Section II. S; is the 
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sum of squares of deviations of the Y;; from the least squares regres- 
sion line of y on x for the sample O;, . 

G, can be computed from the samples. For large values of the 
%,, Ga, in repeated sets of samples O,, O.,---, Ox, is approximately 
distributed according to the chi-square law with (K — 1) degrees of 
freedom if the hypothesis H, is true (i.e., if o,2 = oo? = --- = ox’). 
(See pp. 151-152 of reference 3.) 

The small-sample distribution theory of G, (or 4,4) is complicated 
and will not be presented. The large-sample theory just stated is suf- 
ficient for problems which have been encountered in the Educational 
Testing Service which involve several hundred cases per sample. 

If the value of G, is significantly large as determined from the 
chi-square distribution with (K — 1) degrees of freedom, we con- 
sider this as evidence that the samples do not support the hypothesis 
H,. This means that the standard error of estimate computed from 
the individual samples varies significantly from sample to sample. 

If G, is significantly large, i.e., if the standard error of estimate 
in the different samples varies too much from sample to sample, we 
stop the testing procedure. If, however, G, is not significantly large, 
we may regard hypothesis H, as not having been contradicted by the 


samples, and we may assume o,? = o”, = --- = o,? and proceed to the 
next stage and test the hypothesis (Hz) that the slopes of the regres- 
sion lines are equal, i.e., that 6, = B. =---= Br. 


The likelihood ratio criterion, 4; , for testing H, is defined as the 
ratio of two quantities ms/M; , where M; is the maximum of expres- 
sion (2) with respect to a; , f., 0? (k= 1,2,---,K) after putting 


a1? = 922 = --- = ox? =o’; and mz is the maximum of (2) with respect 
to o, , 8, o? (k=1,2,---, K), after putting o,? = o.?=---= ox =o? 
and £, = fp. =-:+ = Bx =f. (It should be observed that m, is iden- 


tical with M,;). Denoting — 2 log. 4s by Gz, the expression for Gz is 
as follows: 


is 1 * 
=N/I ls |= _ s. |}. 5 
Gs | og vs og. ve k (5) 


where S;, is defined as before in (4) and 


[= > run | 





: K mk k=1 i=1 
S=3 5 ui’?— re ‘ (6) 
si it D> > Xi? 
k=1 i= 


The quantity S is the sum of squares of deviations of all Y’s in all. 


hy 
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samples from their respective least-squares regression lines of y on x 
where the regression lines of all samples are forced to be parallel. 

For large values of the n,, Gs, in repeated sets of samples 
O,, O.,-+-, Ox is approximately distributed according to a chi-square 
distribution with K — 1 degrees of freedom if hypothesis H; is true 
(i.e., if the variances are equal and if the slopes are equal). Signifi- 
cantly large values of G; on the basis of. this chi-square distribution 
may be taken to indicate that the samples cannot be regarded as hav- 
ing come from populations with parallel regression lines (we are as- 
suming, of course, that G, was found to be not significantly large). 
In other words, if G, was not significant, a significantly large value 
of Gz indicates excessive variation in the slopes of the sample regres- 
sion lines from sample to sample so that the samples must be consid- 
ered as coming from populations with non-parallel regression lines. 

It should be pointed out that when hypothesis H; is true, there 
is an exact small-sample theory test for test H;. This test is a Snede- 
cor F test and would be defined (see pp. 170-171 of reference 3, and 
Chapter 12 of reference 2) by putting 


(N—2K)(5— 5 s,) 
F= en, (7) 
«—0 (25) 


and using Snedecor’s F tables with (K — 1) and (N — 2K) degrees 
of freedom, respectively. In case G; is significant, we do not proceed 
further with testing. However, if G; is not significantly large, we con- 
sider this as evidence that the samples support the hypothesis H; that 
the regression lines in the population have the same slope. We then 
proceed to test hypothesis H; , i.e., that the intercepts of the popula- 
tion regression lines are all equal, assuming that the variances about 
the population regression lines are equal and that the slopes of the 
population regression lines are equal. H, is essentially the hypothesis 
that the population regression lines are identically the same line, giv- 
en that the lines are at least parallel and have equal standard errors 
of estimate. 

The likelihood ratio 4, for testing H- is defined as the ratio 
mc/M-, where M, is the maximum of (2) with respect to 0, 6, o? 
(k=1,2,---, K) after putting 8, = f. = ---= Bx and o,? = o,? = 
+» = ox”; and m, is the maximum of (2) with respect to Ms B, o, 
after — a, —-a,—---mar, fi = ps —=.--= Be and o,? = o.? = 
--t= 7. (It should be ue that mz is sdeatioad with M,.) Denot- 


ing — 2 log. 4c by Gc , the expression for Gc is given by 











HAROLD GULLIKSEN AND S. S. WILKS 99 
1 - A: 
Go=N | tog.( = ) —loe.( = 8 | ‘ (8) 
where S is defined in (6) and 


K my K 2 
[> > tutes + SD Mere v' | 





as K mm K ’ k=1 i=1 k=1 
S=[3 3 ue+S mye? |e, . . 9) 
= ¢=1 k=1 [> > Lei? =" > mas? 
k=1 i=1 k=1 


The quantity S is the sum of squares of deviations of all Y’s from the 
least-squares regression line obtained by pooling gross scores of all 
samples together into a single grand sample of gross scores. 

For large values of the m, Gc, in repeated sets of samples 
O,, O., +++, Ox, is approximately distributed according to the chi- 
square law with K — 1 degrees of freedom when H, is true, i.e., when 
all samples are drawn from populations in which the standard errors 
of estimate are equal and in which the regression lines have the same 
slopes and same intercepts. Thus, if Ge is not significantly large, 
(assuming that neither G, nor Gs has been found to be significantly 
large) then we may conclude that the samples are from the same basic 
population, or at least from populations having regression lines with 
the same slopes and the same intercepts, and having the same stand- 
ard errors of estimate. 

We remark that for small samples one may set up a Snedecor 
F-ratio test, namely, 


(N —K—1)(S—S) 
F= ‘ (10) 
(K~—1)8 


and use the Snedecor F-tables with (K—1) and (N—K—1) degrees 
of freedom (see pp. 170-171 of reference 3 and Chapter 12 of refer- 
ence 2). 

In considering hypotheses H,, Hz, and H,, one may wish to 
test them all simultaneously as the first step. If the simultaneous test 
should turn out to be non-significant, then there would be no point in 
proceeding to test H,, H;, and He, separately. One could conclude 
from non-significance of a simultaneous test that the samples could 
be regarded as coming from populations having equal standard er- 
rors of estimate and having the same regression line. If the compre- 
hensive test were significant, then one could proceed to test H., Hz, 
and H, as outlined in the preceding pages. 
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Let Hg denote the over-all hypothesis, 4; the likelihood ratio test 
criterion for testing it, and G; = —2 log. 4s. More specifically, we 
may say that: 


2 — — 


Hs is the hypothesis that o,2 = o.? =--- = 0,73; a, =a, =--- = Oo; 
and B. = fp. =.--= f. 


The likelihood ratio criterion js for testing Hs is defined as the 
ratio ms/M, , where M; is the maximum of (2) for variations of o,?, 


O27, +++ 5 OK} 01, Oo, °***, OK; Br, Bo, ++, Bx, and ms is the maximum of 
(2) for variation of o?, a, and f after putting o,? = o. =---=a7 = 
o*; a, =a, =---—a, =a; and f, = f. =--- =f, = B. (Actually Ms 


is identically the same as M, obtained in testing H,, and msg is iden- 
tically the same as mz, obtained in testing H,.) Denoting —2 log. ds 
by Gs, we find 
a 
Gs = Ga + Gp + Go. 

For large samples, Gs is approximately distributed according to 
the chi-square law with 3(K — 1) degrees of freedom when Hs is 
true. Thus if Gs is non-significant, one may regard the samples as 
having come from populations in which the standard errors of esti- 
mate are equal and in which the regression lines are identically the 
same line. If Gs is significant, we then proceed to test H,, Hz, and 
H, successively. 


III. General Case 
Having considered the bivariate case in which there is one de- 
pendent and one independent variable, we shall now turn to the gen- 
eral case in which we have one dependent variable and several inde- 
pendent variables. These independent variables may be designated 
Xi(h = 1,---, A). 
We shall use the following additional notation: 


Xixi designates the h-th independent variable 
score for the i-th individual in the k-th sam- 
ple 


O, designates the k-th sample and consists of the 
sets of scores (Xixi, Xoxi, +++ » Xuni, Yui)s 
4=1,2,°-+,% 


1™ 
Xi. =— S Xn, designates the mean X-score for the h-th in- 
N, i=1 
dependent variable in the k-th sample 
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nei = Xini — Xn. Aenotes the x deviation score for the h-th in- 
dependent variable score for the i-th individ- 
ual in the k-th sample. 

1 Xai denotes the grand mean over all samples for 


xX 


= 


zl 
iMe 


2X' nx. = Xix. — Xp.. denotes the deviation of the mean of the h-th 
independent variable in the k-th sample from 
the grand mean of the h-th independent vari- 
able in all samples. 


In this case, we consider O; as a sample of ™ sets of values 
(Xaxi ’ Xoxi gees X yki 9 Yi), (2 =] ’ 2 g °° 5 Nx) drawn from a popu- 
lation in which Y;; is considered to be normally distributed with mean 


H ? 
ox + S Bu Xii and with variance o,?. Thus O,; is regarded as a sam- 
h=1 
ple from a multivariate population in which Y is normally distributed 
with standard error of estimate o, around the regression plane of Y 


H 
on X,, X2,-°-- , Xx. having equation Y = a, + 5 Bu X, such that the 


h=1 
values of the X; are “fixed.” 
The probability element of Y;; is 


1 A 
1 - — (Yxi-Qe- & BriXnzi)? 
20;2 h=1 


vies j dYii, (11) 


and the probability element of all the Y;; is 





H 


K 1 — - 1S ¥ (vpi-en~ E CneX ani)?/o08 K m 
va ( ) “ -@ WT WdyY,;. (12) 
kar \ 270%? kay i=t 
The three generalized forms of the hypotheses H, , Hz , Ho which 
we wish to test may be defined as follows, denoting them by H,", H;’, 
H,*, respectively. 


(a) H,° is the hypothesis that o,2 = o.2=--- = o;? (i.e., that the 
standard errors of estimates are iene in all popula- 
tions) regardless of the values of the a, and Bx . 


(b) Hz" is the hypothesis that Bhi = Bro = Bas = +++ = Bre (i.e. 
that the regression planes in the various populations are 
all parallel) assuming that H,* is true (i.e., that o,? = 
a2? = ++» = ox”), regardless of the values of the o;, . 
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(c) Ho* is the hypothesis that a, = a, = --- = ag assuming H;"° 
is true (i.e., that the regression planes, which are as- 
sumed parallel, are in fact identically the same plane). 


The procedure for finding the likelihood criteria G,*, Gz*, and 
G,’* for testing hypothesis H,°, Hz’, and H-’ is similar to that by which 
the criteria G,, Gz, and G, are obtained for testing hypothesis H, , 
Hz, and Hz, respectively. 
The criterion G,* for testing hypothesis H,* is as follows: 


1 ¢* K 1 
Gt =N log,| —> s, |- > % log.| — sy | , (13) 
N k=1 ¥() 





k=1 
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Ss. = , (14) 
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i=1 i=1 i=1 i=1 
A ae nk Ld nk Ld 
D> Yori Linieri D XL oxi o 0 oD LexiXuni (15) 


in i=1 i=1 i=1 


nk nn nk 


La 4 
D> Yutur Dd Muito DS LoxiXuni so 6 ee 


i=1 i=1 i=1 i=1 








and Axoo is the minor formed from 4A, by deleting the first row and 
first column. 

For large values of 1%, G,*° is approximately distributed accord- 
ing to the chi-square law with (K — 1) degrees of freedom if hypothe- 
sis H,° is true, i.e., if the standard errors of estimate of the K popu- 
lations are all equal. 
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Special case. For the case of two predictors (H = 2), the formula 
for G,° is given by (13) where 


> Y*xi > Yuitxi D> YriTori 
int ina inn 
ne nm ne 
> Yitus > xi D CiriTexi 
é=1 é=1 i=1 


ne nk 


bi) 
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i=1 i=1 i=1 











Sf = (16) 
ne Ld 
DM LY Mrs Loni 
i=1 i=1 
nm TM 
D> CaniXexs >> Vans 
i=1 é=1 








and when H,° is true, G,* is approximately distributed according to 
chi-square law with (K — 1) degrees of freedom when each of the 
samples is large. 

In the case of three predictors, S,° would be given as a 4th-order 
determinant divided by a 3rd-order determinant. These determinants 
can be obtained by adding another row and column in the numerator 
and denominator in formula (16) for S,". 

If the conclusion from testing with G,* is that H,° is false, i.e., 
that the standard errors of estimate for the various populations from 
which the samples come can not be regarded as identical, it has been 
shown that the samples are heterogeneous with respect to their stand- 
ard errors of estimate and further tests are inapplicable. If, how- 
ever, the test with G,* shows that the standard errors of estimate for 
the samples may be regarded as homogeneous, we may then proceed 
to the next step and test the hypothesis H;* that the regression planes 
are completely parallel. We do not test separately the hypothesis that 
the slopes in one direction are equal and then that the slopes in an- 
other direction are equal. We simply make one over-all test for “par- 
allelism of regression planes” in the various samples. The likelihood 
criterion G;* for testing the hypothesis H,° is 


Of | og.| ~ s- | —to8.| = 3 s. || (17) 
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where S,° is defined as before and S* is defined as follows: 


_. 
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and A,, is the minor formed from A by deleting the first row and first column. 








19) 
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For large samples G;° is approximately distributed according to 
the chi-square law with H(K — 1) degrees of freedom when H;’° is 
true, i.e, when the population regression planes are parallel and when 
the population standard errors of estimate are all equal. A small- 
sample F test similar to that given in (7) can be set up in this gen- 
eral case, the number of degrees of freedom for the two sums of 


h K K 
squares (s—35,) and > S,* being H(K—1) and (VN — K—HkK), 
k=1 k=1 


respectively. 
For the case of two predictors, (H = 2) S;° in formula (17) is 


given by (16) and S* is as follows: 
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For the case of two predictors G;° is approximately distributed 
according to a chi-square law with 2(K — 1) degrees of freedom when 
H;’ is true. 

With three predictors (H = 3), expression (17) holds for G3’, 
with S;*° and S* each given as the ratio of a 4th-order determinant 
to a 3rd-order determinant. For this case, Gs* will have 3(K — 1) 
degrees of freedom. 

If G;* is significantly large, then we must consider that the sam- 
ples are from populations having regression planes which are not 
parallel. If G;* is not significantly large, we may regard the samples 
as having come from populations with parallel regression planes. We 
may proceed to test the hypothesis H,* that the regression planes are 
identically the same plane. 
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The likelihood criterion G,* for testing H,” is given by 


Ge =N { tog.( 5") —108.( 5+) , (21) 


where S° is defined as in (18) and S° is given by 
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For large samples the quantity G,* is approximately distributed 
according to a chi-square law with (K — 1) degrees of freedom, when 
H,’ is true; i.e., it is true if the standard errors of estimate are equal 
for all populations and if the regression planes are identical for all 
populations. A small-sample F test similar to that given in (10) 
can be constructed from the sums of squares (S* — S*) and S*. The 
numbers of degrees of freedom for the two sums of squares are 
(K— 1) and (N — K — B), respectively. 


For two predictors, G,* is given by (21) , where S* is given by 
(20) and S* is given by 
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G-* will have (K — 1) degrees of freedom. Correspondingly, 
the expression for G,* in the case of three predictors (H = 8) is 


similar. S* and S* are each given as the ratio of a 4th-order determi- 
nant to a 3rd-order determinant, and the number of degrees of free- 
dom for G,* is still (K — 1). 

As in the case of one predictor, we can test H,*, H;*, and H,* 
simultaneously. In this case the hypothesis Hs* we would be testing 
is that o,2 = o.? = --- = og?; a, = a = :-- = ax; and Bn = Be 
=.--=Bu (h=1,2,---, H). The test criterion G;* for testing H;" 
is given by 


 : 1 
G;* =N log. ( one s*) a bs naloge| Ts S;" (25) 
N k=1 %, 
or 
G;* = Peg <i G;* = G-’. (26) 


When 4H;’ is true, G;* is approximately distributed according to 
the chi-square law with (H + 2)(K —1) degrees of freedom. 


IV. Illustrative Problem* 


TABLE 1 
Mean and Standard Deviation of Veterans and Non-Veterans in College Grades, 
Scholastic Aptitude Test—Verbal, and Scholastic 
Aptitude Test—Mathematical 














Non-Vet Vet 
Mean St.Dev. Mean St.Dev. 
g 75.776 6.350 75.104 6.065 
SAT-V 564.788 86.135 536.138 93.306 
SAT-M 593.679 79.718 575.875 83.507 
N 694. 5381. 


Table 1 gives the means and standard deviations of college grades 
(g), Scholastic Aptitude Test—Verbal (V), and Scholastic Aptitude 
Test—Mathematical (M) for a group of 694 non-veterans and a group 
of 531 veterans in an eastern college. It will be seen that there is 
only a tenth of a standard deviation difference in the mean grade of 
the veteran and non-veteran groups. There is, however, a very much 
larger difference (3 or 4 tenths of a standard deviation) between the 


*These data, from a forthcoming study of the college performance of vet- 
erans by the Educational Testing Service under a grant from the Carnegie Cor- 
—— were made available by Dr. Norman Frederiksen and Dr. W. B. 
Schrader. 











HAROLD GULLIKSEN AND S. S. WILKS 111 


means of the two groups in the two sections of the Scholastic Apti- 
tude Test. If allowance is made for the ability difference between 
veterans and non-veterans indicated between the two sections of the 
Scholastic Aptitude Test, are the veterans — relative to their ability 
— doing significantly better or worse than the non-veterans? In other 
words, can the same regression plane of grades on SAT-V and SAT-M 
be used for veterans and non-veterans? 

In addition to the means and standard deviations for each of the 
groups, it is necessary to have the matrix of intercorrelations among 
the variables for the veteran group and for the non-veteran group. 
This information is given in Table 2. 


TABLE 2 


Intercorrelations of Grades, SAT-Verbal, and SAT-Mathematical 
for Veterans and Non-Veterans 























Non-Vet 

g V M 
g : ES 3909 .3010 
V .3909 a 1781 

M .3010 1731 5B 
Vet 

g V M 

1 .4130 3292 


Vv 4130 iI 2331 
M 3292 2331 1. 





Testing Hypothesis of Equality of Standard Errors of Estimate 
for Veterans and Non-Veterans 


The first step in testing whether or not the same regression plane 
can apply to both groups is to determine whether or not the standard 
error of estimate of grades from the two predictor variables (SAT-V 
and SAT-M) is essentially the same. The necessary information for 
testing this hypothesis is given in Table 3. The value of the determi- 
nant of the matrix of intercorrelations for veterans and non-veter- 
ans and the value of the first minor of this determinant are shown 
in Table 3. The standard error of estimate is 31.9 for the non-veter- 
ans and 28.4 for the veterans. The sum of the squares is for non- 
veterans 22,137, and for veterans 15,083. Adding these two sums of 
squares and dividing by the total number of cases (1,225) gives 30.4 
as the pooled error of estimate. Since G,* is 2.0395 with one degree 
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of freedom, it is not significant and we may regard the pooled value, 
30.4, as essentially representing the error of estimate both for vet- 
erans and for non-veterans. Since we have a common value for the 
error of estimate, it is now possible to proceed to test the second hy- 
pothesis, namely, that the regression planes are parallel. 


TABLE 3 
Test for Hypothesis of Equality of Errors of Estimate 


| 























Non-Vet | Vet 
A= _-.767,366,783,58 -730,107,115,20 
Ajo== _ -970,086,39 945,664,390 
rr 
= = 31.897,924 28.404,213 
M, — Ago 
S;,° = 22,187.15930 15,082.637,04 
%.— 694 531 
=S,,° = 87,219.79634 
2n,—= 1225. 
2s,° 
——= 30,883,507 | 
=n, | 
Ge 2.0895 | 


Degrees of 
freedom = (K —1) = 


bo me 


Pe 


Testing the Hypothesis that the Regression Planes are Parallel 

In testing the hypothesis that the regression planes are parallel, 
it is necessary to obtain a variance-covariance and a sum of squares 
and cross-products matrix based on pooling deviation scores for vet- 
erans and for non-veterans. In order to do this, we use Tables 1 and 
2 to compute a variance-covariance for each of the two groups. The 
nine terms in each of these two matrices are then multiplied by the 
respective number of cases in order to give the sum of squares and 
cross-products matrix for veterans and for non-veterans. These two 
matrices are then added to give the matrix shown in Table 4. The 
determinant of this matrix divided by its first minor is 37,2438, the 
sum of squares for the error of estimate obtained when deviation 
scores are pooled. The error of estimate corresponding to this sum 
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of squares is 30.402, which is to be compared with the value 30.384 
which was found in Table 3. Computing G;*, we find again that it 
gives a p value greater than .5. Hence we conclude that making the 
regression planes parallel did not result in a significant increase in 
the error of estimate. This increase was only .02. Since the planes 
may be regarded as parallel, we now see if they can be regarded as 
identical. 


TABLE 4 
Testing Hypothesis of Parallel Regression Planes 
g V M 
g 47,519.459 272,495.071 194,284.965 
V 272,495.071 9,771,842.444 1,789,310.201 


M 194,284.965 1,789,310.201 8,113,227.424 


A/A,, = S* = 37,242.9507 
S*/N = 380.402,408 
=S,,*/N = 30.383,507 
N = 1225. 
ee .7164 
Degrees of 


Freedom = (K—1) (H) = 2 
(1) (2) 


| > .50 
| <.70 


Testing Hypothesis of Identical Regression Planes 

In order to test the hypothesis of identical regression planes, the 
procedure is to pool gross scores for the groups involved and to see 
if the errors of estimate obtained when gross scores are pooled are 
significantly greater than those obtained when deviation scores are 
pooled. In order to find the sum of the squares and cross-products ma- 
trix for the pooled gross scores, it is necessary to add the correction 
terms indicated in the formula for G,*. When these terms are comput- 
ed and added to the matrix shown in Table 4, we obtain the matrix of 
sums of squares and cross-products that would have been obtained 
from pooling the original gross scores. This matrix is shown in 
Table 5. The determinant of this matrix divided by its first minor is 
S* and is equal to 37,281. The error of estimate obtained from pool- 
ing gross scores is equal to 30.433 as compared with 30.402, which 
is obtained in pooling deviation scores. Is this increase in error of 
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estimate significant or may we say that the error of 30.433 made 
in pooling gross scores is essentially equivalent to the error of 30.402 
made in pooling deviation scores? The test with G,* again shows that 


TABLE 5 
Testing the Hypothesis of Identical Regression Planes 
47,655.106 278,282.538 197,881.477 


278,282.538 1U,018,768.634 1,942,757.806 
197,881.477 1,942,757.806 8,208,584.533 


A — 
— = S* = 37,280.7564 
Avo 
S*/N= 30.438,271 
S*/N= 30.402,408 
N= 2e36. 
Gs 1.2467 
Degrees of 
Freedom — (K—1) = 1 
> 2 
# “ 3 


the difference is non-significant, which means that we may regard 
the veterans and non-veterans as having come from populations with 
the same regression plane. 

If the over-all test of H,*, H,*, and H,* had been made first, we 
would compare the smallest possible error shown in line three of 
Table 3 (31.897,924 and 28.404,213) with the error made by assum- 
ing that one regression plane fits both groups, (30.433,271) shown 
in Table 5. Substituting these values in the equation for Gs", we find 
that G;* = 4.0026. This is also given as the sum of G,*, G;* , and G,*. 
This quantity is distributed as chi-square with (H + 2)(K — 1) or 
> 
< 5 
taneous test of H,*, H,;*, and H,* shows that veterans and non-vet- 
erans may be regarded as fitting the same regression plane. 
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TEST RELIABILITY AND CORRECTION FOR ATTENUATION 
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Evidence is cited to show that specificity, or lack of equiva- 
lence, in the comparable forms of tests has a tendency to lower 
the value of reliability coefficients but has no tendency to lower the 
value of observed trait coefficients. This implies that the greater 
the lack of equivalence, the higher will be coefficients corrected for 
attenuation. Errors of measurement are supposed to reduce the 
magnitude of observed trait coefficients. Since specificity does not 
lower the correlation between two tests and since the split-half 
and equivalent-form reliability coefficients treat specificity as er- 
ror, it follows that these two coefficients cannot legitimately be 
used in Spearman’s correction-for-attenuation formula. 


At the time that Spearman first published his formula designed 
to adjust observed correlation coefficients for the attenuating effect 
of errors of measurement, Pearson and others were skeptical of its 
validity, mainly because some corrected coefficients turned out to be 
greater than unity. Most of this skepticism disappeared when Yule 
presented his mathematical basis for the derivation of the formula. 
Later, Johnson (5) furnished rather convincing empirical evidence 
that, when the assumption of independence among errors and among 
errors and true scores is met, Spearman’s formula gives very accur- 
ate results. 

In practice, however, there still exist doubt and uncertainty a3 
to the interpretation that should be given to corrected coefficients. 
This is due to our inability to compute acceptable reliability coeffi- 
cients. Guttman (4), for instance, feels that the split-half coefficient 
underestimates reliability and that corrections for attenuation based 
on this coefficient are fallaciously high. He defines reliability in terms 
of many independent trials of the same test rather than in terms of 
independent trials of “equivalent” forms, and believes that this is 
the interpretation of reliability held by Spearman. 

The main purpose of this paper is to call attention to the fact 
that lack of equivalence in the comparable forms of tests has a ten- 
dency to lower the magnitude of reliability coefficients but has no 
tendency to lower the magnitude of observed trait coefficients. This 
phenomenon has important implications for investigators interested 
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in true relationships, for, unless they give it due consideration, they 
cannot properly evaluate coefficients corrected for attenuation. 


It is generally agreed that lack of equivalence in the comparable 
forms of a test reduces the magnitude of the reliability coefficient, 
and this matter need not be discussed here. The influence that speci- 
ficity has upon observed trait coefficients, however, has not received 
much attention, and it seems desirable to cite a few hypothetical ex- 
amples to illustrate what often occurs under actual testing conditions. 


Suppose that we have two comparable forms each of a reading 
test, an arithmetic test, and an intelligence test. To facilitate dis- 
cussion, we shall assume that each form of both the reading test and 
the arithmetic test correlates .70 with each form of the intelligence 
test, and that each form of the reading test correlates .50 with each 
form of the arithmetic test. We shall further assume comparable- 
form reliability coefficients of .99 for all three tests. 

The two forms of the reading test will be designated R, and R., 
the two forms of the arithmetic test A, and A., and the two forms 
of the intelligence test J, and J. . 

The first form of the reading test, R,, has a correlation of .70 
with the first form of the intelligence test, J, .. Now suppose that the 
specificity in the two forms of the reading test is increased by sub- 
stituting some arithmetic items from A, for a number of the reading 
items in R,. These arithmetic items have the same correlation with 
the intelligence test as the reading items they displace; hence this 
new form of the reading test will have the same correlation as the 
old form with each of the forms of the intelligence test. In other 
words, the observed trait coefficient has not been changed by in- 
creasing the lack of equivalence between the two forms of the read- 
ing test. The correlation between R, and-R., however, will now be 
less than the .90 that it was previously. 

One may investigate the effect that a pronounced lack of equiv- 
alence in the comparable forms of tests has upon corrected coeffi- 
cients by substituting the first form of the arithmetic test for the 
first form of the reading test and the second form of the arithmetic 
test for the second form of the intelligence test. The two “compar- 
able” forms of the “reading” test will now consist of one form of the 
arithmetic test, A, , and one form of the reading test, R., and the two 
“comparable” forms of the “intelligence” test will consist of one form 
of the intelligence test, J, , and one form of the arithmetic test, A. 

The reader is warned against the assumption that all this is ab- 
surd, for in actual practice, when attempts are made to measure cer- 
tain personality traits or obscure psychological traits, the two “equiva- 
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lent” forms of a test may have no more in common than a reading test 
and an arithmetic test. 

The reliability of the “reading” test is now .50 (r between A, and 
R, ), and the reliability of the “intelligence” test is .70 (7 between I, 
and A, ). The four trait coefficients between the tests (A,J,, A,Az, 
R21, , R2A,) will have values of .70, .90, .70, and .50. The two reliability 
coefficients have a geometric mean of nearly .60, and the four trait 
coefficients have a geometric mean of nearly .70. The corrected co- 
efficient obtained by dividing the geometric mean of the four trait co- 
efficients by the geometric mean of the two reliability coefficients has a 
value of about 1.17. 

A similar situation must exist whenever Spearman’s formula gives 
a corrected coefficient greater than unity. There must be lower correla- 
tion within tests than between tests—reliability coefficients must be 
lower than observed trait coefficients. 

The above illustration indicates clearly that the main cause of un- 
duly high corrected coefficients is the lack of equivalence in the split 
halves or the comparable forms of tests. Ordinary errors of measure- 
ment may sometimes arrange themselves in such a manner that the 
correlation within tests will be lower than the correlation between 
tests, but this is not likely to occur very often. 

The effect that lack of equivalence in the comparable forms has 
upon reliability and trait coefficients may be explained in terms of a 
general factor, group factors, and specific factors. It may happen that 
the correlation between two different tests is due entirely to the fact 
that they measure the same general factor. It is then possible to alter 
the group factors and the specific factors measured by the various 
forms without altering the correlation between tests. One form may 
measure different group and specific factors than another form, but as 
long as they measure the same general factor, both forms will corre- 
late the same with a second test. Thus it is possible for lack of equiva- 
lence to change the size of reliability coefficients without changing the 
size of observed trait coefficients. 

A corrected coefficient indicates the correlation between what is 
common to the two forms of one test and what is common to the two 
forms of a second test. If the specificity in the forms of both tests is of 
such high degree that what they have in common is restricted to a 
general factor, and if this general factor happens to be the same in both 
instances, as it frequently is in educational and psychological meas- 
ures, then correction for attenuation will consist chiefly of determin- 
ing how highly a general factor correlates with itself. Under such 
conditions we should expect a corrected coefficient to approach the 
limit 1.00 (perfect correlation) as the specificity in the forms of the’ 
tests increases. 
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When the two forms of a test lack equivalence to a considerable 
extent, the correlation between them is more aptly called a trait co- 
efficient than a reliability coefficient. Using the geometric mean of a 
pair of trait coefficients in Spearman’s attenuation formula is not a 
highly commendable statistical procedure. 

The appropriate reliability coefficient to use in Spearman’s formu- 
la is the coefficient that treats as error those things, and only those 
things, that have a tendency to attenuate observed coefficients. Spe- 
cificity does not attenuate an observed coefficient and should not be in- 
cluded as error in computing a reliability coefficient. On the other hand, 
the two tests from which a trait coefficient is computed are usually 
given with an intervening interval of time. Instability, therefore, should 
be included as error. The reliability coefficient that treats instability, 
but not specificity, as error is the one that Cronbach (2) calls the co- 
efficient of stability. It is obtained from many independent trials of the 
same test administered at different times. The intervening intervals 
should be of about the same length as the interval between the adminis- 
tration of the two different tests from which the trait coefficient is ob- 
tained. 

Since we are not able to achieve independent trials of the same 
test, the coefficient of stability must remain a theoretical conception. 
It still is useful, however, for we can compare reliability coefficients 
computed with it and make an estimate of how well they meet require- 
ments. 

The split-half coefficient treats specificity, but not instability, as 
error. In computing an observed coefficient, instability, but not specifi- 
city, is included as error. If these two (instability and specificity) are 
of equal importance in their attenuating effect, then the split-half co- 
efficient is acceptable for use in Spearman’s formula. It is the opinion 
of the writer, however, that specificity, in general, lowers the value of 
reliability coefficients more than all other factors combined and that 
the split-half coefficient seriously overestimates attenuation. Neverthe- 
less, it should not be overlooked that in measuring personality traits 
and certain psychological traits influenced by emotions, the instability 
factor might lower an observed coefficient to such an extent that the 
split-half coefficient would underestimate attenuation. 

The conclusion reached in this paper, then, is that from the stand- 
point of correction for attenuation, all the reliability coefficients that 
we are able to compute, with the exception of the test-retest coefficient 
and in rare instances the split-half coefficient, underestimate the re- 
liability of our tests. In most cases, corrected coefficients based on the 
comparable-form or split-half reliability coefficient will be much too 
high. 
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ON THE IDENTIFIABILITY OF PARAMETERS IN THUR- 
STONE’S MULTIPLE FACTOR ANALYSIS* 
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In econometric literature a parameter in a theoretical model 
has been called identifiable if it can be uniquely determined in terms 
of the joint probability distribution of the observed variables. In 
this paper the identifiability of parameters in four different factor 
analysis models is considered. The last of these four models corre- 
sponds to Thurstone’s factor analysis. In Sections 7 and 11, the 
possibility of a statistical testing of the models is discussed. Section 
10 deals with the problem of actually determining the parameter 
r (the number of common factors) in terms of the probability dis- 
tribution of the observed variables. 


1. Introduction 


Among the problems which Thurstone discusses in his books on 
multiple factor analysis are the questions of uniqueness of the com- 
munalities (11, 73-77, and 12, 307-311) and the uniqueness of the 
factor loadings (11, 155, and 12, 334). Instead of the term “unique- 
ness,” which Thurstone uses, I shall in this paper use the term “iden- 
tifiability,” introduced by staff members of the Cowles Commission 
for Research in Economics. The question of the identifiability of 
parameters in theoretical models has been discussed by several writ- 
ers in the econometric field before 1940, particularly by Ragnar 
Frisch. The main contributions to the formalization, generalization, 
and rigorous analysis of the problem were made by Haavelmo (2, 
Chapter 5), Koopmans and Rubin (6), and Hurwicz (4), all writing 
in the econometric field. 

The theoretical model considered in this paper has some analogy 
to a particular case of the econometric model considered by Koop- 
mans and Rubin in the paper just referred to. The fundamental 
equation (2.1) in the present paper has its counterpart in the reduced 
form in the econometric model, but in the econometric model the vari- 
ables X are observed variables, while in the psychometric model they 
are latent variables. Because of this difference the analogy men- 
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tioned is of no help in the discussion of identifiability in the factor 
analysis model. Theorem 5.1, however, brings in another analogy be- 
tween the two models. Again there is the difference that the iden- 
tifiability of the matrix A in factor analysis depends upon assump- 
tions about a certain minimum number of zero elements in each col- 
umn of A, the locations of which are not known a priori, while in 
the econometric model the identifiability of the corresponding ma- 
trix depends on assumptions about given elements of the matrix being 
zero, or about linear relationships between given elements of the 
matrix. Because of this difference the discussion of the identifiabil- 
ity is more complicated in the factor analysis model than in the econ- 
ometric model.* 


2. The theoretical model of multiple factor analysis 
We shall use the following notation: 


8; = score in test j, 
Xm = score in common factor m , 
Qjm = factor loading of common factor m in test 7, 
u; = unique part of s; (i.e. specific factor + error), 
n= number of tests, and 
7 = number of common factors. 


We shall introduce the following row vectors: 


S= [s81,82,°°° pad, 
X = [21 , Xe ,°**, Xr], and 
U = [ty , Ue ,°*+, Un) ; 


and the matrix: 
11 » M2 5°***, Qir 
A= Goi » Moe »°** » Der 
! Ani » Un2 »*** » Onr 
M,Q, and D will be used to denote the population covariance mat- 
rices of the vector variables S, X , and U, respectively. The trans- 
pose of a matrix A will be denoted by A’. The diagonal elements of 
the matrix M—D will be called communalities, and the diagonal ele- 
ments of the matrix D will be called uniquenesses. Thurstone uses 
these terms after standardization of the variables S. 
*A more detailed comparison between the two models is given in a joint 


paper by Koopmans and myself (7). I am also indebted to Professor Koopmans 
for valuable suggestions and criticisms regarding the present paper. 
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The variables S are the only observed variables. The variables 
X and U are not observed and will be called latent variables. 

Our model will be given by the following specifications, all of 
which refer to the hypothetical population, not to a sample. For the 
population mean (mathematical expectation) we shall use the usual 
symbol E . 


S’=AX'+ UV’. (2.1) 
E(X)=0 and E(U)=0. (2.2) 

The set of variables X is stochastically independ- 
ent of the set of variable U. (2.3) 
D is diagonal and different from 0 .* (2.4) 
X and U are normally distributed. (2.5) 
A has rank r. (2.6) 
Q is nonsingular. (2.7) 


Each s; is correlated with at least one of the 
other s’s. (2.8) 


r is the smallest number of variables X which is 
compatible with the joint probability distribution 
of the observed variables and_ specifications 
(2.1) through (2.8) (2.9) 


Each column of A contains at least r zeroes. (2.10) 


A normalization rule fixing the units of the vari- 
ables X and a rule fixing the order of the columns 
of A. (2.11) 


Consider the matrix consisting of all the rows of A which have 
a zero in the mth column. Let this matrix be denoted by A,,. Let 
the number of rows in the matrix A, be p» . Let Ani denote the sub- 
matrix of A, constructed by deleting the ith row of A,,. Using these 
notations we shall formulate the final specification of our model. 


The rank of each of the matrices A»; (m= 1, 
2,°°°, 754=—1,2,---, Dm) is 7T—1. (2.12) 
Specification (2.2) does not represent any loss of generality. 


*A matrix (or vector) is said to be equal to zero when each of its elements 
is equal to zero. It is said to be different from zero if at least one of its ele- 
ments is different from zero. 
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Specifications (2.4) and (2.5) imply that the variables U are mu- 
tually independent. The assumption that D is different from zero 
is made for convenience in order to avoid discussion of a case of no 
practical interest. It is equivalent to the assumption that M is non- 
singular. Specification (2.8) implies that no row of the matrix A 
contains only zero elements. 

The model considered represents an interpretation of Thur- 
stone’s multiple factor analysis. This presentation of the specifica- 
tions is, however, somewhat different from that of Thurstone, partly 
because we have chosen the specifications in such a way that a rigor- 
ous discussion of identifiability should not become too complicated, 
and partly because we have tried to formulate specifications which 
are in a sense minimum conditions for identifiability. 

We have included in our model Thurstone’s assumption of sim- 
ple structure, but not the assumption that the factor loadings are 
non-negative. Thurstone says about the postulate of non-negative 
factor loadings (12, 341), “But it must be understood that this is a 
postulate of much psychological investigation in a particular field 
and that it does not constitute a restriction upon the principle of 
simple structure, which is applicable either with or without the spe- 
cial restriction that the factor loadings shall be positive or zero.” 

If all parameters A, Q, and D in our model are given nu- 
merically, we shall talk about a structure. The structure includes 
completely specified distributions of the latent variables and a set 
of equations with numerically given coefficients connecting the ob- 
served and latent variables. A structure thus is a particular realiza- 
tion of the model, and the model is the set of all structures compatible 
with the given specifications. 

From Equations (2.1) and (2.2) we obtain 


E(S) =0 (2.13) 
and 


M=E(S'S) = E[(AX’ + U’) (XA' + U)] =AE(X'X)A' 
+ AE(X'U) + E(U'X)A'’+E(UU). 


From (2.3) this reduces to 
M=AQA’+D. (2.14) 


From (2.1) and (2.5) we conclude that the variables S are 
jointly normally distributed. 


A given structure T = {A,Q,D} generates one and only one joint 








OLAV REIERS@L 125 


probability distribution P(S) of the observed variables S. On the 
other hand there may possibly be several structures 7 generating the 
same distribution P(S). If two or more structures generate the same 
joint probability distribution of the observed variables, the struc- 
tures are said to be equivalent. If a parameter has the same value in 
all equivalent structures it is said to be identifiable. In other words, 
a parameter is identifiable if it can be uniquely determined from a 
knowledge of the joint probability distribution of the observed vari- 
ables. If a parameter is not identifiable, no consistent estimate of the 
parameter will exist. 


3. Consideration of the model defined by specifications 
(2.1) through (2.8) 


For this model we shall in the following use the notation Model 
(2.1-8). We shall study this model as a preliminary step in the study 
of the complete model given in Section 2. 


Let a distribution P(S) with a covariance matrix M be given, 
and suppose that we have a set of three matrices A, Q, and D 
satisfying Equation (2.14) and the following conditions: 


A has the same rank as its number of columns, (3.1) 
Q is a symmetric and positive definite, (3.2) 
D is diagonal, non-negative, and different from 0. (3.3) 


Let X and U be two sets of normally distributed variables with the 
joint probability density 





V (det Q) (det D) 
e 


-14(XOX'+UDU") 
’ 





n+T 


where 7 is the order of the matrix Q. These variables satisfy Speci- 
fications (2.2) through (2.7), and when we define the set of vari- 
ables S by S’ = AX’ + U’, the variables S will have the same joint 
distribution as the given variables S. Hence we have the following 
theorem: 


Theorem 3.1 The set of structures equivalent in Model (2.1-8) 
and generating a distribution P(S) with covariance matrix M is 
given by the set of all matrices A, Q, and D satisfying Equation 
(2.14) and Conditions (3.1) through (3.3). 
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When the matrix Q is positive definite, the matrix AQA’ is posi- 
tive semidefinite (when r < n) or positive definite (when r = n). 
Hence we have: 


M — D is positive definite or semidefinite. (3.4) 


Suppose inversely that (3.4) holds good. Now any positive def- 
inite or positive semidefinite matrix can be transformed by a co- 
gredient transformation to any other positive definite or semidefinite 
matrix of the same rank. If the rank of M—D is 7, we can therefore 
find a non-singular matrix H, such that 


oe Q 0 , 
M D — H, | 0 0 H, > 
where Q is a symmetric, r-rowed, positive definite matrix. Let FH: 


denote the matrix consisting of the 7 first columns of H, and let H; 
denote the matrix consisting of the last m — r columns of H,. Then 


we can write 
ss Q 0 a,’ 
M—D=[H2H:] | 0 0 | H,’ x 


which reduces to 
M rai D = H.QH.’. 


This gives a solution of (2.14) with A = H., and it is immediately 
seen that this solution satisfies Conditions (3.1) through (3.3). Hence 
we have 


Theorem 3.2. Let the covariance matrix M and the order r of 
the matrix Q be given. Then a necessary and sufficient condition for 
the existence of a solution of Equation (2.14) satisfying Conditions 
(3.1) through (3.3) is that there exists a matrix D such that Condi- 
tions (3.3) and (3.4) are satisfied and the rank M— Disr. When a 
solution exists, there exist infinitely many solutions, since the matrix 
Q can be any symmetric, r-rowed, and positive definite matrix. 


Next we shall prove 


Theorem 3.3. Suppose that Model (2.1-8) contains a structure 
T where r= s <n and where D is non-singular. Then the model 
contains an infinity of structures equivalent to T where r=s +1 
and where D is non-singular. 


Let A, Q, D be a structure where r = s and the determinant 
of D #0. Then M — D is of rank s, and it contains at least one 
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s-rowed principal minor which is different from zero. Let P be such 
a minor and let mi; — d; be a diagonal element not contained in P. 
Let us consider the (s + 1)-rowed minor P, which we obtain when 
adding the ith row and the ith column of M — D to the minor P. Let 
us change d; to d;*, where 0 < d;* < d;. This change will increase 
the value of P, and, since it was zero before the change, it must be 
positive after the change. Let D* be a diagonal matrix which we 
get from D by changing the single element d; to d;*. Then D* is non- 
negative and non-singular and M — D* is positive semidefinite and 
of rank s + 1. Using Theorems 3.1 and 3.2, Theorem 3.3 follows. 


Let the diagonal elements d,, d., --- , d, of the matrix D be 
interpreted as Cartesian coordinates in an n-dimensional space 
which we shall call D-space, and let the covariance matrix M be 
fixed and non-singular. Then there exists an n-dimensional region 
in D-space where D is non-negative and different from zero and 
where M — D is positive definite, and there exists an (n — 1)-dimen- 
sional region in D-space where D is non-negative and different from 
zero and where M — D is positive semidefinite and of rank n — 1. 
(Cf. 9, p. 15). Hence there exist structures in Model (2.1-8) where 
r =n, and structures where r = n — 1, all of them generating the 
given covariance matrix M . Hence r is not identifiable in this model, 
but may have at least two values, n and n — 1. And for each of 
these values of 7 there exists an infinity of possible matrices D. 
Again for each possible D there exists an infinity of possible ma- 
trices A and Q. We are thus very far from identifiability in this 
model. 


If we drop the specification that the matrix A has rank 7, any 
value of r which is greater than n — 2 is possible. The lack of iden- 
tifiability in a model of this kind was pointed out in 1919 by G. H. 
Thomson (10), who gave numerical examples of different structures 
generating the same covariance matrix of the observed variables. 


4. Identifiability of the uniquenesses in a model defined 
by Specifications (2.1) through (2.9) 


We may reformulate Specification (2.9) as follows: Consider 
the set of all equivalent structures in Model (2.1-8) which generate 
a given distribution P(S). 7 is the least number of variables X oc- 
curring in any of these structures. A third way of formulating Speci- 
fication (2.9) is the following: For a given covariance matrix M 
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consider the set of all solutions of Equation (2.14) satisfying con- 
ditions (3.1)-(3.3). r is the least value of the order of Q occurring 
in any solution in this set. A fourth way of formulating the speci- 
fication will be: For a given covariance matrix M consider the set 
of all matrices D such that (3.3) and (3.4) hold. rv is the minimum 
rank of M — D within this set. 


It follows immediately from any of these formulations that 7 is 
always identifiable in Model (2.1-9), and from the considerations in 
Section 3 it follows thatr = n—1. 


We shall next consider the identifiability of the uniquenesses, 
which are the diagonal elements d; of the matrix D. Walter Leder- 
mann has shown (8) that for the determination of the d’s we have 


k,=4 (n—1r) (w—r1r +1) 


equations, which are generally independent. We shall make this 

statement more precise. Let the elements of the matrix M be inter- 
k(k+1) 

preted as Cartesian coordinates in a —— space, 


which we shail call M-space. The k, equations will be independent at 
all points of M-space except for a subspace of lower dimensionality. 


< : > 
We have k, “ m according as 7 ri Yn, Where 


%, —+4 (Qn+1—vV8mr+1). 


The k, equations are not linear. In the case when r = 7 we 
shall therefore usually not have one solution only, (except in the 
case when n = 3, r = 1) but a finite number of solutions. In order 
to be admissible these solutions must satisfy Conditions (3.3) and 
(3.4). These conditions may or may not exclude all solutions except 
one (15). ; 

When r > r, we shall generally expect D to be non-identifiable, 
but it may be identifiable for some sets of values of the elements of 
M. When r < 7, we shall expect D to be identifiable, but it will fail 
to be identifiable for some sets of values of the elements of M. 

Even though we do not know much about the exact conditions 
for identifiability of the uniquenesses, we may state that the iden- 
tifiability is in principle subject to statistical test. It has been shown 
that the identifiability of parameters in any model is in principle 
subject to statistical test (7). 
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5. The set of equivalent structures in Model (2.1-9) 


We shall find the set of equivalent structures in the case when 
D is identifiable. Two equivalent structures must then have the same 
D. Let the structures T = {A, Q, D} and T* = {A’, Q*, D} be 
equivalent. Then 


M— D=AQA'= A* Q°A”. (5.1) 
We can write this equation in the form 
A ‘ . * * s 
t | gtaravy=[4", | orcas Au"T, (5.2) 
An An 


where A; and A;,* represent the upper square parts of A and A’, re- 
spectively. Since A is of rank 7 we can without loss of generality 
assume that A; is non-singular. From (5.2) we get 4 equations be- 
tween the submatrices. We shall use two of these equations, 


A, QA, = A;* Q* A*’ (5.3) 
and 
An QA; = An" Q* Ar”. (5.4) 
Setting 
V=A,'Ay, (5.5) 
V is a square, 7-rowed, non-singular matrix, and we get from (5.3) 
Q=VQ* Vv’. (5.6) 
From (5.5) we get 
A/*=ArV. (5.7) . 
Using (5.6) and (5.7) we obtain from (5.4) 
Ay*=AnV. (5.8) 


Equations (5.7) and (5.8) can be written together as one single ma- 
trix equation 


A*=AY. (5.9) 


We have now shown that (5.6) and (5.9) are necessary condi- 
tions for the equivalence of the structures 7 and T”. Inversely, suppose 
that the structure T = {A, Q, D} generates the covariance matrix 
M , and let A* and Q” be defined by (5.6) and (5.9), where V is any 
square r-rowed, non-singular matrix. Then it follows that (5.1) 
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holds good and that A* and Q* satisfy conditions (3.1) and (3.2). 
Hence we have the following theorem: 


Theorem 5.1: The set of all structures {A*, Q*, D} equivalent in 
Model (2.1-9) to the structure {A, Q, D} is given by the set of all 
matrices A* = AV and Q* = V"Q(V’)“, where V is any square, 
r-rowed and non-singular matrix. 


Among the matrices A* there may possibly be some matrices 
where one or more columns contain only one non-zero element. This 
would mean that the factor occurs only in a single test and will 
therefore not be a common factor. We have, however, not excluded 
such a case by our specifications. Strictly speaking the x’s need not 
represent common factors. It should be noted, however, that no x 
can be independent of all the other x’s and at the same time occur 
in one test only. For, suppose that the factor x, occurs in the jth 
test only. Then we may include a;,,2%», in u; instead of in s; — u;. 
After this change our specifications will still hold. good with 7 re- 
placed by — 1. But this contradicts the specification defining 7 and 
is therefore impossible. 


6. Identifiability in Model (2.1-11) 


Specifications (2.1) through (2.10) are never sufficient for the 
identifiability of A, for if we have one matrix A satisfying these 
specifications, they will still be satisfied if we multiply each column 
by an arbitrary factor and perform an arbitrary permutation of the 
columns. In order to take care of this indeterminancy we have intro- 
duced Specification (2.11). The normalization rule fixing the units 
of the variables X may be formulated in various way in terms of the 
elements of the matrix Q or in terms of the elements of the matrix 
A. We shall give three examples of possible formulations. 


The variance Qmm Of 2m is equal to 1.00 (m=1,2,---,7). (6.1) 


= mj =1, (m=1,2,---,7). (6.2) 
j=1 
D |@mj]=1, (m=1,2,---,7r). (6.3) 
j=1 


Also the rule fixing the order of the columns of A can be formulated 
in various ways. 


The following theorem is evident: 
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Theorem 6.1. A necessary and sufficient condition for A to be 
identifiable in Model (2.1-11) is that all matrices A* equivalent to 
A in Model (2.1-10) can be written in the form AV, where Visa 
square non-singular matrix which contains exactly one non-zero ele- 
ment in each row and exactly one non-zero element in each column. 


The addition of Specification (2.10) may possibly make the ma- 
trix D identifiable even if it is not identifiable in Model (2.1-9). We 
shall, however, not enter into any discussion of this question, but 
shall confine the discussion to the identifiability of A in the case 
when D is identifiable in Model (2.1-9). 

We shall now introduce a geometrical representation of the ma- 
trix A. (Cf. 12, Chapter 3). Before we do that we shall define a 
few concepts of multidimensional geometry. The set of points in 
n-dimensional space whose Cartesian coordinates satisfy a single lin- 
ear equation will be called a hyperplane. The intersection of two or 
more hyperplanes will be called a flat. If it is of p dimensions, it 
will be called a p-flat. A one-flat is a straight line, a two-flat is a 
plane, and an (n—1)-flat is a hyperplane. A figure bounded by hyper- 
planes will be called a polytope. The simplest polytope in n-dimen- 
sional space is bounded by +1 hyperplanes. It is called a stmplez. 
A simplex in two-dimensional space is 2 triangle, in three-dimen- 
sional space a tetrahedron. 

We shall now consider the geometrical representation already 
announced. Let the ith row of the matrix A be denoted by a;. Let 
us choose a Cartesian coordinate system C in r-dimensional space, 
and let g; be a vector whose components in the coordinate system C 
are the elements of the row a;. Let e,, @,-::, e, be unit vectors 
along the coordinate axes in the system C. Then 

Gi =X ij e;. (6.4) 
g=1 
Let us next choose a new Cartesian coordinate system C*, and let 
€,", €2", +++, e,” denote unit vectors along the axes in the system C. 
Let vj, be the kth component of e; in the coordinate system C*, so 
that 


€; => Vix ex’. (6.5) 


1 
Let aix," be the kth component of g; in the system C ,, so that 


Gi =D Aix” ey". (6.6) 


k=1 
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Substituting (6.5) in (6.4) we obtain 


Hix’ =D Ai; Vix, (6.7) 


j=1 
which can be written in matrix form 
A°=AYV. (6.8) 


To each transformation V of the coordinate system corresponds 
one and only one matrix A* in the set of matrices which are equiva- 
lent in Model (2.1-9) to the matrix A. Inversely, if a matrix A* 
equivalent to A is given, V is uniquely determined by (5.5), and we 
can find a coordinate system C* such that the ith row of a; of A gives 
the components of the vector g; in the coordinate system C*, for 
i=1,2,---,m. The axes of the coordinate system C* are uniquely 
determined by the equations 


rT 
e;* — > vik Cx, 

k=1 
where v* are the elements of the matrix V-. This gives a possible 
coordinate system, since we do not assume that the coordinate sys- 
tems are orthogonal. There is thus a one-to-one correspondence be- 
tween the set of all possible Matrices A* equivalent in Model (2.1-9) 
to a given matrix A , and the set of all possible Cartesian coordinate 
systems C with the same origin as C. Specification (2.10) can now 
be reformulated thus: 


Each coordinate hyperplane contains at least r row 
vectors of the matrix A. (2.10b) 


The problem of the identifial:ility of A , given that of D, is now 
seen to be equivalent to the question of whether the coordinate hy- 
perplanes are uniquely determined by Condition (2.10b). The length 
of the units along the coordinate axes and the numbering of the co- 
ordinate axes are not determined by (2.10b), but will be determined 
by (2.11). 


It it possible to get a geometric representation where the num- 
ber of dimensions is reduced by one. This can be done by intersect- 
ing all test vectors g; or their extensions forward or backward by a 
hyperplane H, which does not pass through the origin. If we in- 
clude the points at infinity in this hyperplane there is a one-to-one 
correspondence between the test vectors g; and the intersection 
points h; in the hyperplane. These intersection points will be called 
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test points. The intersections between the hyperplane H and the co- 
ordinate hyperplanes in r-dimensional space will again be called coor- 
dinate hyperplanes in (7 — 1)-dimensional space. They will be bound- 
ing hyperplanes of an (r — 1)-dimensional simplex, which we shall 
call the fundamental simplex. The intersection between two or more 
coordinate hyperplanes will be called a coordinate p-fiat if it is of 
dimensionality ». We can now give a second reformulation of speci- 
fication (2.10). 


Each coordinate hyperplane in the space H 
contains at least 7 of the test points h;. (2.10c) 


The problem of the identifiability of A is seen to be equivalent to the 
question of whether the coordinate hyperplanes in the space H are 
uniquely determined by Condition (2.10c). 


We shall first show that Specification (2.10) in a certain sense 
is a minimum requirement for identifiability of the matrix A. This 
will be expressed in the following theorem: 


Theorem 6.2 Let the number r in Specification (2.10) be re- 
placed by r — 1. Then the matrix A is not identifiable. 


Proof: Suppose as before that the upper square part A; of A is 
non-singular and consider the matrix 


eh Re Pees t 
o=| 4 [4 lc, | - 


where C,, = Ay,A;'. From Specification (2.8) follows that each row 
of A must be different from zero. Since A;" is non-singular, also 
each row of C must be different from zero. Since r = n — 1, the 
matrix C,;; must contain at least one row. Let c,,¢.,--:, ¢, be the 
elements of the first row of C,,. Without loss of generality we can 
assume that c, = 0. Post-multiplying the matrix C by the matrix 





1 Co Cc, 

Cy Cy Cy 

¥, = 0 , ae 0 
{ O. +@..aosa's 1 | 





we get the matrix 
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i Ti ae oe 
Ci Cy Cy 
c= 0 ati ia. te 0 
0 syst mies | 

q 1 D «is os 0 | 








This matrix has at least y—1 zeros in each column. If at least one of 
the elements c2, ¢;,--:, ¢, is different from zero, the transformation 
V, is not of the form V. The only case which is still in doubt is when 


c, is the only element in the first row of C,, which is different from 
zero. 


In this case we postmultiply C by the following matrix: 





aoe acer 0 0 
a 2 oe ee o 8 
Ss Be ee 
ie were a a = 
/-: wre = * 
where p.,--- , Pr-+, P, are arbitrary. We then get the matrix 
| ‘a, ae 3c oe 
Ss eee 0 0 
the? he. ae 1 0 
a See Dia Dr 
a eer a 0 ®” / 








and in this matrix again each column contains at least r — 1 zeros. 
Choosing both p, and at least one of the other p’s different from zero, 
the transformation matrix V, is non-singular and not of the type V. 
This completes the proof of Theorem 6.2. 


Theorem 6.8. When D is identifiable in Model (2.1-9), a nec- 
essary and sufficient condition for A to be identifiable in Model 
(2.1-11) is that any square r-rowed minor of A which is of rank 
r — 1 is contained in one of the matrices An. 
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This theorem may be reformulated in terms of the geometric 
representation in the space H as follows: 


Theorem 6.3b. When D is identifiable in Model (2.1-9), a neces- 
sary and sufficient condition for A to be identifiable in Model 
(2.1-11) is that any hyperplane containing at least r test points is 
a coordinate hyperplane. 


When the condition is satisfied the 7 coordinate hyperplanes are 
uniquely determined as the r hyperplanes each of which contains at 
least 7 test points. On the other hand, if there exists a hyperplane 
which is not a coordinate hyperplane, but which contains at least r 
test points, then there exists an equivalent structure where this hy- 
perplane is a coordinate hyperplane. This proves the theorem. 

Suppose that a coordinate (r— 3)-flat contains r — 1 test points. 
Then any hyperplane H’ through this (7 — 3)-flat and any other test 
point will contain at least r test points. Suppose that there is at least 
one test point not lying in one of the two coordinate hyperplanes 
whose intersection is the (7 — 3)-flat. Then a hyperplane H’ through 
this test point and the (r — 8)-flat will contain r test points, and H’ - 
will be different from any of the coordinate hyperplanes. Suppose 
next that a coordinate (r — 38)-flat contains r or more test points. 
Then any hyperplanes through the (r — 3)-flat will contain r test 
points. In this case we thus have an infinity of hyperplanes H’ each 
of which contains at least 7 test points and such that each H’ is dif- 
ferent from any coordinate hyperplane. 

The condition of Theorem 6.3 will therefore generally exclude 
the case of a coordinate (r — 3)-flat containing 7 — 1 test points and 
it will always exclude the case of a coordinate (7 — 3)-flat containing 
r or more test points. But according to Thurstone these excluded 
cases are what we may expect in practice, and they are even desir- 
able (12, 335, Condition 4). The results of the present section are 
therefore not practically useful. We want a model where the coordi- 
nate hyperplanes are characterized not only by the requirement that 
each of them shall contain at least 7 points, since we shall expect 
many other hyperplanes to contain r or more points. A natural 
assumption is that each coordinate hyperplane shall be overdeter- 
mined by the points in it, i.e., if we take away any single point in the 
coordinate hyperplane, the remaining points will still determine the 
hyperplane. This is just what we have expressed algebraically in 
Specification (2.12). But before we consider the problem of identi- 
fiability of the parameters in the complete model we shall consider 
the possibility of testing statistically the specifications. 
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7. On the possibility of testing the specifications 


If any joint probability distribution of the observed variables 
can be generated from a structure belonging to the model we are con- 
sidering, the model cannot be tested statistically. Let us first con- 
sider Model: (2.1-9). We can test Specification (2.5), taking for 
granted that Specifications (2.1) and (2.3) hold good. Specifica- 
tion (2.8) can also be tested, since it is an assumption about the ob- 
served variables. No further testing of the model is possible, since 
any covariance matrix M satisfying Specification (2.8) can be gen- 
erated by a structure in the model. 

In Model (2.1-11) it is in principle possible to test Specification 
(2.10) assuming that Specifications (2.1) through (2.9) hold good. 
And in Model (2.1-12) it is in principle possible to test Specifications 
(2.10) and (2.12), taking for granted that Specifications (2.1) 
through (2.9) hold good. We shall consider the latter test only. The 
set of admissible hypotheses is now given by all structures satisfying 
Specifications (2.1) through (2.9), and the hypothesis to be tested 
includes all structures satisfying Specifications (2.1) through (2.12). 

Let the matrix A belong to a structure in Model (2.1-9) and let B 
be any matrix of the form B = AV, where V is square and non-singu- 
lar. Let us consider the matrices A, , A2,-:- , A, considered before, 
and let B,, be a submatrix of B consisting of the rows of B which 
have the same position in B as the rows of Am have in A. Then 
By» = AmV, and since V is non-singular, the rank of B,, must be the 
same as the rank of A,,. Let B,; denote the submatrix of B which 
we get when we delete the 7th row of B,,. Then Bni = AmiV, hence 
the rank of B,,; is the same as the rank of An; . 

Suppose now that Specifications (2.10) and (2.12) are satisfied 
for the matrix A. Since A, contains a column consisting of zerus 
only, its rank cannot be greater than r — 1, and since it contains 
submatrices of rank r— 1, its rank must be exactly r — 1. Hence we 


get 
The rank of each B,, isr —1. (7.1) 
The rank of each By; isr—1. (7.2) 


The matrix A,, is supposed to contain all rows with zeros in the mth 
column. Henée, if we add to A, a row of A which is not contained 
in A,,, the resulting matrix must be of rank r, and from this we 
again concltde: 
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The addition to B, of a row of B not contained 
in B,, increases the rank to 7. (7.3) 


In addition to the matrices B, , B,,---, B,, there may possibly exist 
other submatrices of B with r columns, satisfying Conditions (7.1) 
through (7.3). If such matrices exist we shall denote them by 
Bri, 00> » By. 

Setting W = V-1, W is a square, r-rowed, non-singular matrix 
and A = BW. Let W,, denote the mth column of W. Since the mth 
column of A,, consists of zeros only, we have B,,W, = 0. Since the 
columns of W are linearly independent we can state 


The set of right nullspace vectors* of the ma- 
trices B, , B. ,--- , B, contains r linearly independ- 
ent vectors. (7.4) 


Suppose inversely that a matrix B satisfies Conditions (7.1) 
through (7.4). Among the right nullspace vectors W; of the ma- 
trices B; we choose r which are linearly independent, and we form a 
matrix W the columns of which are these r nullspace vectors. 
Then it is evident that the matrix A = BW satisfies Specifications 
(2.10) and (2.12). We shall formulate our results in the following 
theorem: 


Theorem 7.1. Let B be any matrix AV, where V is a square, 
non-singular matrix. A necessary and sufficient condition for the 
matrix A to satisfy Specifications (2.10) and (2.12) is that the ma- 
trix B satisfies Conditions (7.1)-(7.4). 


Suppose now that the population covariance matrix M is known, 
and that the matrix D is identifiable. Then the matrix M — D can be 
determined and we can find a matrix B such that BB’ = M — D, for 
instance by the diagonal method of factoring (12, 101). Next we can 
examine if the matrix B satisfies Conditions (7.1) through (7.4). 
If it does satisfy these conditions, we know that there exists a struc- 
ture in Model (2.1-12) generating a distribution P(S) with a covari- 
ance matrix M. If B does not satisfy Conditions (7.1) through 
(7.4), Model (2.1-12) must be rejected. If Specifications (2.1) 
through (2.9) are taken for granted we shall have to reject Specifi- 
cations (2.10) and (2.12) or at least Specification (2.12). 

So far we have discussed what conclusions we can draw about 
the specifications when we know the population covariance matrix. 


*A right nullspace vector of a matrix is a vector which vanishes after a 
premultiplication by the matrix. 
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When we do not know the population covariance matrix, as is the 
case in practice, but know only a sample, it follows, however, from 
the foregoing considerations that it is possible, at least in point of 
principle, to test Specifications (2.10) and (2.12). 


8. Reformulation of Specifications (2.12) in terms of 
assumptions about non-zero elements 


Instead of Specification (2.12), Thurstone makes assumptions 
about non-zero elements in the matrix A (11, 156 and 12, 335). We 
shall reformulate Specification (2.12) in a similar form. 

Let matrix B be considered as a function of some of its elements 
while the other elements are constants. Let the elements of B which 
we regard as variables be denoted by z,, 22,-°+:, 2, and let Z denote 
the vector 

Z = [15 oy°**y Spl. 


The matrix B regarded as a function of the variables Z will be de- 
noted by B(Z). Any minor determinant in B will be a polynomial 
in the variables Z or identically equal to zero. If it is not identically 
equal to zero it will be equal to zero only in a set of measure zero in 
Z-space. The rank of the matrix B will be a function of Z. The 
maximum of the rank B when Z varies will be called the rank of the 
matrix function B(Z). Evidently this rank is the rank of B in the 
whole Z-space, except for a set of measure zero. 

Let Y be a vector whose components are all elements of B which 
are not components of Z. So far we have considered Y as constant. 
Let us now consider the minimum rank of B(Z) when Y varies over 
all possible values. This minimum will be called the minimum rank 
of B(Z). 


Lemma 8.1. B(Z) attains its minimum rank when Y=0. 


Proof: Any minor determinant of B is a polynomial in the vari- 
ables Z. The coefficient of each term in the polynomial is either 1 
or —1 or a homogeneous function of the variables Y of degree one or 
higher degree. Hence, if a coefficient in the polynomial vanishes for 
some value of Y, it vanishes when Y = 0. From this follows that if a 
minor determinant of B vanishes identically in Z for some value of 
Y, it will vanish identically in Z when Y = 0. This proves the lemma. 


Lemma 8.2. Let B be a matrix with r columns and a number of 
rows which is at least equal tor. When Y = 0, a necessary and suf- 
ficient condition for the rank of B(Z) to be equal tor is that the 
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following condition is satisfied forq—1,2,---,r: For each set of q 
columns of B there exist at least q rows, each of which has a variable 
z in at least one of the q colunms. 


Proof: We shall first prove the necessity of the condition. Sup- 
pose that the condition is not satisfied. Then there exists for some q 
a set of g columns of B which contain elements of Z in at most q—l 
rows. Any r-rowed minor of B may be expanded in a Laplace expan- 
sion according to the q columns. Since all g-rowed determinants in 
these gq columns are zero, each r-rowed minor of B must be zero iden- 
tically in the variables Z. Hence the rank of B(Z) is less than r. 
This proves the necessity of the condition in Lemma 8.2. 

We shall next prove the sufficiency of the condition. We shall 
prove this by induction. The condition is evidently sufficient when 
r=1. The sufficiency statement will therefore be proved in the gen- 
eral case if we can prove that it is true for a matrix B with r columns 
provided that it is true for any matrix with less than r columns. We 
thus have to prove that if the condition is satisfied for a matrix B 
with r columns and if Lemma 8.2 is true for any matrix with less 
than 7 columns, then the rank of B(Z) must be 7. In order to prove 
this we shall assume that the rank of B is r — 1 and show that this 
leads to a contradiction. 

Let us consider the submatrix of B obtained by deleting the rth 
column of B. According to our assumptions this submatrix is of 
rank r — 1, and there exists a product bj, 1 bi,,2 ++: bi, ,-a where each 
factor is az. By an interchange of rows if necessary we can obtain 
that 7, =1,%=2,---i,,—=r—1. Let us next consider the product 
Bir Doo +++ Dy-s,r-1 Dir Where i 2 r. Since the rank of B is r — 1 by as- 
sumption, we must have b;, = 0 when i 2 r. We now have the fol- 
lowing picture of the matrix B. 
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R 


0 
Column r contains at least one z, and since bi, = 0 wheni27,a2 
must occur in one of first r — 1 rows. If b,..,, is not a z we can always 
by an interchange of rows and the corresponding interchange of 
columns obtain that b,_,,, will be a z, while the elements in the main 


diagonal which are known to be 2’s still will be situated in the main 
diagonal. Consider now the product 


Das Doo sag Onant Deas Dijr-a 


where i 2 r. Since B is of rank r — 1, all these products must be 
zero; hence we conclude that b;,,. = 0 when i 2 1, since all other 
factors in the product above are known to be z’s. Columns r — 1 and 
r contain z’s in at least two rows. If necessary by an interchange of 
the r — 2 first rows and the corresponding interchange of columns, we 
can obtain that columns r— 1 and r will contain a z in the row r — 2. 
Hence the determinant 


Dp-29-1 Dy-2,r 
Dae Brae 


will not be identically equal to zero. Because B is of rank r — 1, the 
product of this determinant and },; Dez «++ Br-s,r-3 Di»-2 = 0 when 
~72r. Hence b;,,. — 0 wheni 2 r. Carrying on this argument we 
finally conclude that by, = 0 wheni 2 randk=—1,2,-:-,7. This 
results contradicts the assumption that B contains 2’s in r different 
rows. The assumption that the rank of B is r — 1 must therefore be 








OLAV REIERS@L 141 


false, and we conclude that the rank of B is r. This completes the 
proof of Lemma 8.2. 

Let K, denote the matrix which we obtain from A, when we 
delete the mth column (which consists of zeros only), and let Kyi 
denote the matrix which we obtain from A,,; when we delete the mth 
column. We can now state 


Theorem 8.3. When Specification (2.12) is satisfied, each of the 
matrices K,, satisfies the following condition forq=1,2,---,r—lL: 
For any set of q columns of K,, there are at least q+1 rows each of 
which has a non-zero element in at least one of the q columns. 


Proof: When Specification (2.12) is satisfied each matrix K,,; 
is of rank r —1. Applying Lemma 8.2 to the matrix K,,; , we con- 
clude that it satisfies the condition of this lemma with r replaced by 
ry — 1. If there exists a K,, for which the condition of Theorem 8.3 
is not satisfied, it will be possible to remove a row of K, such that 
we obtain a matrix K,,; which does not satisfy the condition of Lem- 
ma 8.2. But we have shown that this is not possible. Hence Theorem 
8.3 is proved. 

By a similar reasoning the following converse of Theorem 8.3 
can be proved. 


Theorem 8.4. Suppose that each of the matrices K, satisfies the 
following condition forq=—1,2,-:-,r—1. For any set of q col- 
umns of K,, there are at least q+1 rows each of which has a non-zero 
element in at least one of the q columns. Then Specification (2.12) 
is satisfied everywhere in the space generated by the non-zero ele- 
ments of A, except for a set of measure zero. 


9. Identifiability in the complete model. 


We shall begin by stating a theorem which is similar to Theo- 
rem 6.1. 


Theorem 9.1 A necessary and sufficient condition for A to be 
identifiable in Model (2.1-12) is that all matrices A* which are equiv- 
alent to A in the model defined by Specifications (2.1) through (2.10) 
and (2.12) can be written in the form AV, where V is a square non- 
singular matrix which contains exactly one non-zero element in each 
row and exactly one non-zero element in each column. 


Algebraically the condition for identifiability of A may be ex- 
pressed in the following equivalent ways. 
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Theorem 9.2. When D is identifiable in Model (2.1-9), a neces- 
sary and sufficient condition for the identifiability of A in Model 
(2.1-12) is that A does not contain other submatrices than A,, Az, 
.., A, satisfying Conditions (7.1) through (7.3), with A substituted 
for B. 

Theorem 9.3. When D is identifiable in Model (2.1-9), a nec- 
sary and sufficient condition for the identifiability of A in Model 
(2.1-12) is that a matrix B = AV (where V is non-singular) con- 
tains exactly r submatrices B,, satisfying Conditions (7.1) through 
(7.4). 


Let us call a hyperplane in (rv — 1)-dimensional space over-deter- 
mined by a set of points lying in the hyperplane if the hyperplane is 
still determined by any point set which we obtain from the first point 
set by removing one point, in other words if none of the sets of re- 
maining points is situated in an (vr — 8)-flat. 

We shall now formulate the condition for identifiability of A in 
terms of the geometrical representation. 


Theorem 9.2b. When D is identifiable in Model (2.1-9), a nec- 
essary and sufficient condition for the identifiability of A in Model 
(2.1-12) is that each hyperplane which is over-determined by the 
test points lying in it is a coordinate hyperplane. 


Theorem 9.3b. When D is identifiable in Model (2.1-9), a neces- 
sary and sufficient condition for the identifiability of A in Model 
(2.1-12) is that there exist exactly r hyperplanes which are over-de- 
termined by the test points lying in them and that these hyperplanes 
are linearly independent, i.e. that they do not pass through the same 
point, 

Specification (2.12) may be reformulated as follows: 


Each of the coordinate hyperplanes is over- 
determined by the test points lying in the 
hyperplane. (2.12b) 


The proofs of the geometric formulations 9.2b and 9.3b are now 
obvious. 

Theorems 9.2 and 9.3 may be proved by showing the equivalence 
between the algebraic and geometric formulations. We shall, how- 
ever, give an independent algebraic proof. 

Suppose first that A does not contain any submatrix besides 
A,, Az,-++-, A, which satisfies Conditions (7.1) through (7.3). Let 
A and A* = AV be equivalent in Model (2.1-12). Consider one col- 
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umn of A*, say the first column. Since A”* satisfies Specifications 
(2.10) and (2.12), there must be at least r zeros in the column. Let 
A’*,., be the submatrix of A* which consists of all rows of A* which 
have zeros in the first column, and let A,., be the corresponding sub- 
matrix of A. Denoting the first column of V by V,, we have 


Ary Vi=0. (9.1) 


Because A* satisfies Specification (2.12) and A*,., contains all the 
rows of A* which have zeros in the first column, the matrix A,,., must 
satisfy Conditions (7.1) through (7.3). Since A does not contain any 
submatrix besides A,, A., --: , A, which satisfies Conditions (7.1) 
through (7.3), A, must be identical with one of the matrices 
A,, As, «+, A,r, say An. Hence we have An»V, = 0. Since all 
elements of the mth’column of A, are zero, all elements of V, must 
be zero except Vm,. The same proof can be carried through for any 
column of V, so that each column of V contains only one non-zero 
element. Since V is non-singular, it must contain cne and only one 
element in each row. V is therefore of the type V considered in 
Theorem 9.1. 

Inversely, suppose that there exists a submatrix A,., of A which 
satisfies Conditions (7.1) through (7.3) and which is different from 
any of the matrices A,, As,--:, A,. Then there exists a column V, 
such that A,..V, = 0. The column AV, will be different from any 
multiple of any of the columns of A, since its zeros are in different 
places. Hence the column V, must contain at least two non-zero ele- 
ments. Let us now form a matrix V, taking V, as the first column 
and letting each of the other columns contain exactly one non-zero 
element, in such a way that V is non-singular. Then A* = AV and A 
are equivalent in Model (2.1-12), and the transformation matrix V 
is not of the type V. This proves Theorem 9.2. 

When A,, Az,---, A, are the only submatrices of A satisfying 
Conditions (7.1) through (7.3), the matrix B will contain only r 
submatrices satisfying Conditions (7.1) through (7.4). Conversely, 
if there exists a submatrix A,,, which satisfies Conditions (7.1) 
through (7.3), and which is different from any of the matrices 
A,,A.,,-:-,A,, then there exists at least 7+1 submatrices of B satis- 
fying Conditions (7.1) through (7.4). From these remarks Theorem 
9.3 follows. 

Theorem 9.2 expresses the condition for the identifiability of 
A in terms of A itself. This theorem can therefore not be used to find 
out whether A is identifiable or not if we do not have a priori infor- 
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mation about A sufficient to decide whether the condition is fulfilled 
or not. Theorem 9.3 on the other hand makes it possible to examine 
if the condition is fulfilled before A is determined and without any 
a priori information about A . 

Let us finally consider the identifiability of Q when D and A are 
identifiable. Two equivalent structures must now have the same D 
and A. Let T= {A,Q, D} and T* = {A, Q*, D} be any two equiva- 
lent structures. Then Equations (5.5) and (5.6) must be valid with 
A*=A. This gives V = / and Q = Q*. This result may be stated 
as follows: 


Theorem 9.4. When D and A are identifiable Q is also identifi- 
able. 


10. Remarks on the determination of r 


We have seen that r is always identifiable in a model containing 
Specifications (2.1) through (2.9). Hence 7 can always be deter- 
mined if M is known. But the actual determination of r when the 
population covariance matrix is known is difficult. A still more diffi- 
cult problem is the estimation of 7 when only a sample is known. 

I shall mention some results which may be useful in determin- 
ing 7, although they apply only to special cases. 


Theorem 10.1. Necessary and sufficient for r to be equal to one 
is that both of the following conditions are satisfied: 


(a) All tetrad differences are zero, 1.e., 


Mii Mit | 
Mj Mri | 
when i,j,k, and 1areall different. 

(b) 
Miz M5, 


agg = Mj; 
Mik 


0 


IIA 





for all sets of values of i, j}, and k which are all different. 


Condition (a) is due to Spearman and is well known. It is 
usually presented as the only condition, ignoring Condition (b). That 
Condition (b) is necessary follows from Spearman’s formula, 

Miz Mi, 


mj; — d;= “ , (10.1) 
ik 
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and the obvious fact that we must have 
0s M5 — d; = mMj;;. (10.2) 


We shall show that the conditions are also sufficient. When Condi- 
tion (a) is satisfied and D is determined by (10.1), the matrix M —D 
is of rank 1. When Condition (b) is satisfied, (10.2) holds good. 
This implies that the diagonal elements of M — D are non-negative, 
and since all principal minors of order two or higher order are zero, 
M — D is positive semidefinite. From (10.2) also follows that D satis- 
fies Condition (3.3). This proves the sufficiency of the conditions. 

If mi; = 0, either all non-diagonal elements in the ith row and 
column or all non-diagonal elements in the jth row and column of M 
must be zero. This contradicts Specification (2.8). Hence ail ele- 
ments of M must be different from zero when r= 1. 

Condition (a) may also be put in the form that if we ignore the 
diagonal elements, all rows of M are proportional. It is not neces- 
sary to examine whether Condition (b) is fulfilled for all sets of 
values of i, 7, and &. For each value of j it is sufficient to consider 
one i and one k, because that is sufficient to ensure that (10.2) is 
satisfied. Hence we get a modified form of Theorem 10.1. 


Theorem 10.2. Necessary and sufficient for r to be equal to one 
is that each of the following conditions is fulfilled. 


(a) Each element of M is different from zero. 


(b) Excepting the diagonal elements, ali rows of M are propor- 
tional. 


(c) The inequality 





is satisfied for some i and k for each j. 


Consider the inverse M-' of the covariance matrix M. Let us 
change the sign of each row in M™ which has a negative element in 
the first column. Next let us change the sign of each column which 
has a negative element in the first row. If after these sign changes 
all elements of the resulting matrix are positive, we shall say that 
the matrix M- has compatible signs. Using this term we shall state 
the condition for rS n—Z. 


Theorem 10.3 A necessary and sufficient condition for r to be 
less than or equal to n — 2 is that the inverse of the covariance matrix 
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M has signs which are not compatible. 


This theorem is an immediate corollary of a theorem presented 
in one of the writer’s previous papers. (9, Theorem 14). From the 
same theorem we immediately get a necessary condition that r = s, 
where s = n—2. This condition is far from sufficient when s <n—2. 


Theorem 10.4. A necessary condition for r to be less than or 
equal to a number s which is less than n — 1 is that the inverse of 
each (s + 2)-rowed principal minor of M has signs which are not 
compatible. 


As a special case of Theorem 10.3 we get a necessary and suf- 
ficient condition for r = 2 when n = 4. This case was previously 
considered by Kelley (5, 52), Hotelling (5, 54) and Wilson and Wor- 
cester (13). A necessary condition for 7 S 2 when nm = 5 was found 
by Kelley (5, 58). A necessary and sufficient condition for the same 
case was found by Wilson and Worcester (14). . 

Other results have been obtained by A. A. Albert (1, 2). Fora 
given M let p be the minimum rank of M — D when we require D to 
be diagonal, but do not impose the conditions that D shall be non- 
negative and M — D positive semidefinite. Let p’ be the highest order 
of a minor of M which does not contain any diagonal element of M. 
Albert gives necessary and sufficient conditions for the equality 
p =p and gives a method for determining D in the case when this 
equality is fulfilled. The solution arrived at need not necessarily sat- 
isfy Conditions (3.3) and (3.4). 


11. Remarks on the specification defining r 


Specification (2.9) is a mechanical way of defining 7, and at 
first sight it looks impossible that an 7 defined in this way can rep- 
resent any psychological reality. This impression is strengthened 
by Theorem 3.3 which gives the following immediate corollary: If 
there exists a structure in Model (2.1-8) with non-singular D in the 
case when r has its minimum value s, there exist equivalent struc- 
tures in Model (2.1-8) wherer —=s+1,8+2,-:-,n—1,n. 

Nevertheless the data may be such that they give us a reason 
to believe that the value of r defined by Specification (2.9) does ex- 
press some kind of underlying reality. In order to make this clear 
let us consider the following experiment: 

Suppose that a person X constructs a structure in Model (2.1- 
12) in the following way. He chooses Q=I , and constructs a matrix 
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A with n rows and 7, columns, where 
Yo = n=} (Qn +1— V8n + 1). 


He chooses some of the elements in A equal to zero such that the con- 
ditions of Theorem 8.2 are fulfilled when the other elements are dif- 
ferent from zero. He chooses each of the other elements @jm by a 
random drawing from a parent population, for instance a population 
with a rectangular distribution between the limits a@jm and @jm. In 
practice the parent population will have to be discrete. Let us sup- 
pose all possible values of ai; in the population are represented by 
all decimal fractions between the two limits which have a given num- 
ber of decimal places. We shall suppose that the number of possible 
values of each aj; is large, so that the probability of one particular 
value of ai; is small. 

Thereafter X determines the elements d; in the same way by 
random drawings from a parent population. There will be a small 
probability that the structure obtained in this way will not satisfy 
the Specifications of Model (2.1-12). If this should happen X might 
construct another structure or modify the structure he already has 
constructed. Finally X computes the matrix M from equation (2.14) 
and rounds off to a certain number of decimal places. 

He then presents the matrix M to another person Y. He in- 
forms Y about the method used in obtaining M, but does not give 
him any information about the numerical values of the elements of 
A and D or the value of 7). Y now tries to find 7, , using Specifica- 
tions (2.1) through (2.9), and taking into account that the equations 
need not be exactly fulfilled because of rounding-off errors. The 
question now is: Will Y find r = 7, or will he find a smaller value 
of r? 

When rv = 7%, the d; will satisfy 4 (n — 7) (n — 7% + 1) equa- 
tions. If r= 7, — 1, the d; will satisfy 4 (n — ro + 1) (n —% ‘+ 2) 
equations; i.e., in addition to the 4 (n — 7) (n — 7 +1) equations 
which must be satisfied because of the way in which the matrix M 
has been obtained, there will exist n — r. + 1 additional equations 
between the parameters which must happen by chance to be satis- 
fied, and there is only a small probability that this will happen. In 
other words, there is only a small probability that Y will not find 
f= FH,. 

Suppose next that X does not give Y the population covari- 
ance matrix, but a sample from a normal probability distribution 
with covariance matrix M. If the sample is large enough there will 
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still be only a small probability that Y will not estimate correctly the 
value 7. 

From the standpoint of the person Y the argument will be slight- 
ly different. When he finds a value r by assuming Specifications 
(2.1) through (2.9), he will ask whether this is the true value 7) or 
if r, is larger than r. If 7 2 r,, Y will not have any confidence in 
the result, but if r < 7, , the sample which he has got would repre- 
sent an improbable event if 7) > 7 and the sample is large enough. 

Thus we can only have faith in the results of a factor analysis 
if the parameters are overidentified, i.e., if the number of independ- 
ent equations, which determine the parameters in terms of the prob- 
ability distribution of the observed variables, is greater than the 
number of parameters. (Cf. 12, 294). 

In Section 7 we found that the central part of Model (2.1-9) 
cannot be tested. However, if we add the assumption that r <r, , the 
model can be tested; and if we for sufficiently large samples regularly 
find that 7 < 7,,, this will represent a verification of the model. 

It should be pointed out that we have throughout this paper as- 
sumed a fixed battery of tests and a fixed population of individuals 
tested. If we consider the possibility of a partial change of the test 
battery and a change of the population, both the question of the pos- 
sibility of testing the model and the question of identifiability of the 
parameters come in a new light. (Cf. 12, Chapter 16). 
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BIAS AND ERROR IN MULTIPLE-CHOICE TESTS 
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A formula for estimating real scores on a multiple-choice test 
from a knowledge of raw scores is derived. This formula does not 
involve the assumption of a binomial distribution of real scores as 
does the Calandra formula. Other important formulas derived 
show: the variance of real scores in terms of the variance of raw 
scores and the correlation between real scores and raw scores. If 
the variance of real scores (or of raw scores also) is binomial, the 
regression of real scores on raw scores is linear; but, otherwise the 
regression is curvilinear. Yet the linear estimating formula is a 
close approximation to the curvilinear relationship. Factors affect- 
ing the regression of real scores on raw scores and the correlation 
coefficient are: (1) the number of choices per question; (2) the 
number of questions answered; (8) the ratio of the average group 
raw score to the variance of raw scores. 


The major purpose of the multiple-choice type of objective test 
is to eliminate human error in grading or scoring. However, such 
tests are peculiarly subject to a bias and an error not present in non- 
choice objective tests. This situation prevails because students usual- 
ly guess at answers they do not know — in spite of unrealistic instruc- 
tions to the contrary. For the purpose of this paper, we shall as- 
sume that students taking multiple-choice tests will guess at all the 
answers they do not know. As a matter of fact, there is no good rea- 
son why they should not be encouraged to guess. Possibly there is no 
clear-cut line between an informed answer and a shrewd intuitive 
guess. 

Various methods have been proposed for scoring multiple-choice 
tests, and certain theoretically unsound formulas for adjusting test 
scores are widely used. Unfortunately, very few constructive theo- 
retical contributions have been made to this problem, and the best 
contributions have been almost completely ignored. Perhaps the most 
valuable paper on the subject is that of Alexander Calandra, pub- 
lished in 1941 (1). Calandra made Bayes’ theorem of inverse prob- 
ability the basis for scoring multiple-choice examinations. However, 
there is little, if any, evidence that his scoring formula has been 
made use of by educators and professional testers. 
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There is, of course, a long history of empirical research and ex- 
perimentation with scoring methods. The tendency has been to ig- 
nore purely theoretical contributions. It must be granted that any 
and all theory should be tested and checked by empirical methods; 
but on the other hand, it should likewise be conceded that our em- 
pirical research will be facilitated if it takes full advantage of the 
advancements in theory. 

It is not possible in one short paper to review all the important 
contributions to the problem of scoring multiple-choice tests. It is 
interesting to note, however, that over a period of at least 27 years 
there has been in vogue a scoring formula which is not based on 
sound mathematical theory (2, 3). This formula is also widely used 
by professional examination concerns. The formula is: 

W 


inn CG (1) 


where 
S is the adjusted or real score, 
R is the unadjusted or raw score, 


W is the the number of incorrect answers, and 
k is the number of choices per question. 


Other forms of this formula are: 
kW kR—n 


eo ales eo (1a) and (1b) 





S=n— 


where n is the number of questions in a test. 

The above formula is apparently based upon a misunderstand- 
ing of inverse probability. In correlation terminology, it purports 
to be a symbolic statement of the linear regression of real scores on 
raw scores. As a matter of fact, it is nothing more nor less than 
an improper reversal of the formula for the actual linear regression 
of raw scores on real scores. As usually stated, it does not even 
identify S as an estimated score nor do we find anywhere any ref- 
erence to an error of estimate. 

For the most part, in this paper we shall use the symbols, termi- 
nology, and concepts of correlation and regression. 

In a group test situation, there will be N students taking a test 
of n questions and having k choices per question. Unless otherwise 
indicated, we shall assume that all students will answer all questions 
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and that they will guess at random the answers which they do not 
know. Among these students there will exist theoretically two series 
of scores: (1) real scores — the number of answers really known 
and (2) raw scores — the number of questions answered correctly 
whether by knowledge or by guessing. These two series of scores 
may be correlated and certain useful formulas derived. 


In this treatment we shall use the following symbols: 


N= the number of students in a group taking a test. 

n= the number of questions in a test. 

k= the number of choices or alternatives per question, each 
question having the same number of choices. 

S= a real score, i.e, the number of answers really known 
by a student. 

R= araw score, i.e., the number of questions answered cor- 
rectly by a student whether by guessing or by knowl- 
edge. 

W = the number of incorrect answers —=n—R. 


R,S, and W represent the corresponding means of R, S, and 
< 2 
W;e. R=—. 
ai N 
R,S, and W represent values estimated by regression, corre- 
sponding to R, S, and W respectively. 


- s >S - 
Ds= — =< and q.= 1p. 
es R R tie “i 
" eee ee 
n Nn 


In a test of m questions of k choices each, a student can answer 
S of them from knowledge and, on the average, 1/k of those on which 
he is ignorant. 
rage (2) 
k 
or 


Lars (k—1)S 
k ‘oe 


A spurious solution of equation (2) for S yields formula (1), but 


eas (2a) 


the derivation ignores the differences between S and s , R and R ‘ 
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Although S, an individual’s score, cannot be derived from formula 
(2), it is possible to thus determine 5 by summing both sides of the 
equation over the population and by dividing by N,, the number of 
students. The result is that 


~ kR—n 

ma = " 
and dividing both sides by n, 

oe kp, ad. 

SS (4) 


Formula (2a) represents the regression of R-scores on S-scores. n/k 
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FIGURE 1 
Relation between Real Scores and Raw Scores, Assuming that Real Scores 
are Binomially Distributed. 
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we 5 ; , 
is the regression coefficient b,;. 





is a constant and 


bn =H. (5) 
k 
Graphically, this relationship is shown in Figure 1. 

The b,, line is not subject to any pecularities in the distribution 
of S-scores. It is obviously linear and its slope and position are fixed 
by the assumption that the guessing distributions will follow the bi- 
nomial pattern. 

There is an error of estimate of R on S , which may be derived 
as follows: 





_ k—1 
= fe 8 (n “o% ) Ie 
o*+.s — — N ’ (6) 
where f;, represents the number of students having a given S-score; 
n—S represents the number of answers not known by the student; 
is the 


a2 








N represents the number of students in a group; and 


“nq” part of the conventional binomial variance formula: i.e., 1/k is 
the probability of guessing the correct answer. Since k and m are 
constants, formula (6) may be reduced to 


_ (w—S) (k-1) 


S ; (7) 





O'r.s 


and, by substituting the value for S from formula (3), 


n—R_ 
= 


2 
O'r.s 


(8) 





* | | 


This is an important formula, because it can be used to derive the 
correlation between real scores and raw scores and hence the regres- 
sion of SonR. 





Let 
S=a+b, (9) 
”, Tr 
Dae = 10 
— (10) 


jy ee Se (11) 
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By substitution from formula (8), 





ko?, —W 
ae (12) 
Then, from formulas (5), (8), and (10), 
ko?, —W 
Digg FE senertener 1 
(k—1)o?, aad 
and 
— = RW —no?, 
=S— srl — ——_—_——__.. 14 
a=S—b (bie, (14) 
Therefore, 
A PW — 2. a 117 R 
_RW— 12 (ke W) (15) 


” &—Det, (k—De, —- 


Formula (15) was first derived in a different manner by W. A. Hen- 
dricks of the United States Department of Agriculture.* It will be 
shown later that the regression of S on R is curvilinear if the dis- 
tributions of S and R scores are not binomial. Therefore, the useful- 
ness of formula (15) may well be questioned. It would be a useful 
formula only if the straight-line regression approximates the curvi- 
linear regression under the usual circumstances of application — i.e., 
in estimating the real scores of students in class-size groups. It will 
be shown in the latter part of this paper that, in a typical test situa- 
tion, the difference between the straight-line estimate and the curvi- 
linear estimate is not significant in classes with less than about 200 
students. 

Hendricks also showed the relation between the variances of S 
andR. 





, _Bo,—kW 
Sl aia (k —1)2 , (16) 
k—1)?0%, + (k— —S§ 
HD o 1) (n—S) “on 


If R-scores were distributed binomially with a variance of 


o?, = Dr, (18) 


*Available only in personal correspondence with the author. 
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ie R ak ev. 
where p, = — and q, = 1-—»,, Hendricks’ formulas would involve 
n 


only a knowledge of the mean of the R-scores and the number of 
choices per question. By substitution of the values of formulas (3), 
(4), and (18) in Hendricks’ formulas, we have 


kpp—1_ kR—n 











? 9 = oe, (19) 
kp, kR 
kp,—-1 _ kR— 
pp Se ent te es, (20) 
(k—1)p, (kK—1)R 
ke RW —knW 
2= " 21 
° n(k—1)? - 
Qsr Will equal zero, and 
~ (kp,—1)R (kR—n)R 
j= DR_ (UR =2) ae 





(k—1)p, (k—1)R- 
These formulas may also be derived quite simply if we assume 
that the straight regression line passes through the point 
(S=0,R=0), 
as in Figure 1. (No explicit assumption of binomiality of either S 


or R is involved.) Since the line passes through the point (S, R) as 
weil as (0, 0), 


ae. (23) 
R 


and since 


~ kR—n 

= ———_,, 3 

S nae (3) 
(as in 20) 

_ kp,—-1 _ kR—n 

" (k-1)p,  (k-1)R 

Calandra arrived at this same conclusion by assuming that there 

“exists a binomial distribution of examinee knowledge and guessing 

tendency.” His formula, given below, would be mathematically iden- 
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tical with formula (22) if it were assumed that all students would 
answer all questions. 


R 


1+| ~—1 | po. 
Pp 


1 


, (24) (Calandra) 





S (or S) = 


where 


, — the probability that an examinee knows any given item, 

~2—the probability that an examinee will guess at any given 
item, 

; —the probability that an examinee who guesses at an item 
will guess it correctly. 


These p-values are identical with the corresponding values used in 
this paper. For example, 
ee 

Pi Ds $9 k = 1 ’ 

P2—1 (because in this paper it is assumed that all students will 

answer all questions), 

Pe i 4 
By substitution of these equivalents it will be seen that formula (22) 
and Calandra’s formula are identical. And (22) may be rewritten to 
take advantage of Calandra’s p, factor, which takes care of the situa- 
tion where some students do not complete the test: 





. kp,—1]R kR —n]R 
ine =. si —._. (25) 
A similar modification of formula (13) gives 
ko?, —W 
(26) 





i= ee . 
(k —1)0*, — Wade 


The above formula is obtained by multiplying the second form of 


(25) by z and by substituting o*, for Ra-. This procedure would 
qr 
also transform formula (20) to formula (13), although (13) was 
derived in another manner. 
In order to show the relation between the variance of real scores 
and the curvilinearity of the regression of real scores on raw scores, 








C. HORACE HAMILTON 159 


several theoretical tables have been prepared, each table being based 
upon an assumed standard deviation differing from the binomial. 
These tables are not reproduced here but the results are summarized 
below, and some shorter tables (1 and 2) are used for purposes of il- 
lustration. 





























TABLE 1 
Relation between Real Scores and Raw Scores with S = 6 and k = 2 
Real ; Raw Scores 
Scores Oo 2 2 3 4 5 6 if 8 9 10 Total 
10 60 60 
9 201 201 402 
8 302 605 302 1,209 
cf 269 807 807 269 2,152 
6 157 627 939 627 157 2,507 
5 68 314 628 628 314 63 2,010 
4 17 105 261 348 £261 105 17 i,84 
3 38 28 70 116 116 70 23 3 424 
2 & cs ®@ @B 2 28 12 3 0 105 
1 0 1 8 4 4 3 1 0 0 16 
0 0 0 0 1 0 0 0 0 0 1 
Total 0 7 55 266 881 2,014 3,020 2,685 1,072 10,000 
Means (S) 2.25 3.00 3.75 4.50 5.25 600 6.75 7.50 — 
Variances (02, ,) 56 .75 94 112 131 150 169 187 — 
TABLE 2 
Relation between Real Scores and Raw Scores with S = 6 and k = 5 
Real Raw Scores 
Scores Oo tft 2 8 4 5 6 7 8 9 10 Total 
10 60 60 
9 322 80 402 
8 774 387 48 1,209 
q 1,102 826 207 17° “2362 
6 1,027 1,027 385 64 4 2,507 
5 659 8238 412 103 13 1. 200 
4 292 438 274 91 17 2 1,114 
3 89 156 117 49 12 2 424 
2 18 385 31 15 5 1 105 
1 2°S 5 3 1 16 
0 iI 1 
Total 2 24 129 4821,23802,178 2,645 2,107 995 210 10,000 
Means (S) 881.77 2.65 3.58 4.41 5.29 618 7.06 7.94 882 — 


Variances (o?,,) .10 .21 .81 .42 .52 62 .73 83 .98 104 — 
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The method for the following analysis is illustrated in Tables 1 
and 2. These tables show the theoretical correlation between real and 
raw scores in a two-choice and a five-choice test, respectively, each 
of ten questions. In constructing these two tables, it is assumed that 
the distributions of students by real scores are binomial. It is also 
assumed that the average real score in each test is 6, or 60 per cent 
of the 10 questions. The frequency distributions of real scores are 
shown in the right-hand vertical columns of each table. These dis- 
tributions were constructed by use of the following formula. 


i= Cc” pS q™® (10,000) . (27) 
In Table 1, for instance, 
_ 10! , ss 
Jo= rao! (.6)° (.4)7° (10,000) , 
10! 
=F (.6)* (.4)® (10,000) , 
and so on to 
meee 6)2° (.4)° (10,000 
fo =Tor01 © (.4)° (10,000) . 


The second step in the preparation of these tables was the dis- 
tribution of the frequencies for each S-score horizontally by raw 
score, again by means of the binomial distribution. In Table 1, row 
S-10, for instance, the 60 students assumed to have a real score of 
10 were placed in the raw score column 10. They knew all the an- 
swers and checked them correctly. They did not need to guess; but 
note that they represent less than six per cent of the students who 
had a raw score of ten. The 402 students having a real score of 9 
guessed at a 10th question, half of them missing it and half guessing 
it correctly. Thus, 201 of the 402 students make a raw score of 9 and 
the remaining 201 make a raw score of 10. Of the 1,209 students 
having a real score of 8, one-fourth were unlucky and made a raw 
score of only 8; one-half guessed correctly the answer to one addi- 
tional question; and the remaining one-fourth guessed two answers 
correctly and made perfect raw scores. The other rows were filled in 
a similar manner by using a series of binomial proportions, obtained 
by expanding f, (.5 + .5)@°*), 

The means and variances, shown at the bottom of Tables 1 and 
2, are linear functions of R and S. The data of these tables, appro- 
priately analyzed, confirm those formulas of this paper which are 








C. HORACE HAMILTON 161 


based on the binomial assumptions. For instance, the values for 7°,, 
in Tables 1 and 2 are precisely those expected from the formula 


kp,—1_ kR—n 
kp- Ya 


The variances shown in the bottom rows of Tables 1 and 2 were 
calculated by the formula 





or = 


(19) 


W A 
i tae , (28) 
R(k—1) 
which fits the computed values exactly, except for small errors due 
to rounding. Had a much larger N been assumed in the computa- 
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FIGURE 2 : 
Regressions of Real Scores on Raw Scores, Assuming Five Different Vari- 
ances of Real Scores. 
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tions, the errors would have been infinitesimal. Formula (28) is in- 
teresting because it shows that the variance of estimate of S on R 


increases proportionately to S. 

Using the same general method described above, but giving up 
the assumption that S-scores are distributed binomially, five basic 
tables have been prepared to show what happens to the functional 
relationship between S and R when the standard deviation of S 
varies from the binomial.* A test of 30 questions with an average 
real score of 15 (50 per cent) is assumed. However, each of the five 
experimental tables is based upon normal distributions of S-scores 
with standard deviations varying from 1/3 to less than the binomial 
to 2/3 greater than the binomial. Finally, the results of assuming a 
rectilinear distribution are determined. As shown in Table 3, the as- 
sumed standard deviations, on which Figure 2 is also based, are as 
follows: 


(1) 1.83 (1/3 less than the binomial) 
(2) 2.74 (the binomial) 

(3) 3.65 (1/3 more than the binomial) 
(4) 4.56 (2/3 more than the binomial) 
(5) 8.94 (a rectilinear distribution) 


Figure 2 shows that the regression of S on R becomes curvilinear 
as the standard deviation of S-scores departs from the binomial. If 
the standard deviation is less than the binomial, the slope of the re- 
gression curve decreases as R and S increase. On the other hand, if 
the standard deviation of S-scores is greater than the binomial, the 


TABLE 3 
Effect of Increasing the Variance of Real Scores 











Standard Regression Correlation 
Row Variance Deviation Coefficients Ratios* 

ae oe, o, C, b,, Pre a, 7, 

(1) (2) (3) (4) (5) (6) (7) = (8) 

1 3.33 4.58 1.83 2.14 35 .50 18 18 
2 7.50 5.63 2.74 2.38 .67 .50 33 38 
3 13.33 7.18 3.65 2.68 .96 50 AT 48 
4 20.83 8.92 4.56 2.99 1.16 50 59 .58 
5 80.00 14.07 8.94 3.75 1.46 .50 .86 -73 





*E? is used to represent the ratio of the “explained” variance to the total variance; e.g., 
07, — 074.7 
ee SEE A 
oa, 


*It is still assumed that the guessing distribution to questions not known 
will follow a binomial distribution. 
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slope of the regression curve increases as R and S increase. 

Table 3 shows other significant results of this experiment. Col- 
lumn 5, for instance, shows that the linear approximation to the cur- 
vilinear regression, b,,, increases as the standard deviation increas- 
es; and the data of Column 7 and 8 present evidence that there is 
very little difference between E?,, and E?,,, the two variance ratios, 
except in the case of the rectilinear distribution. This means that 
the linear regressions follow the curvilinear regressions quite closely, 
particularly within the middle parts of the distribution of S and R. 

The data of Columns (1) and (2) of Table 3 (except for that of 
Row 5 based on the assumed rectilinear distribution of S-scores) also 
confirm the theory of formulas (16) and (17). This may be verified 
by substituting the Table 3 values for o?p and o’s in formulas (16) 
and (17). 
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FIGURE 8 
Comparison of Curvilinear Regression of Real Scores on Raw Scores with 
the Linear Approximation, Standard Deviation of Real Scores is Assumed to be 
Two-Thirds Greater than the Binomial Standard Deviation. 
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Figure 3 shows, for the case where the standard deviation of 
real scores is two-thirds greater than the binomial, a comparison of 
the actual curvilinear regression of S on R with the linear approxi- 
mation given by the Hendricks formula (13 of this paper). The 
curve of Figure 3 is identical with the 4th curve of Figure 2. The 
curve of Figure 3 is cut off so as to include a range of approximately 
98% of the expected number of raw scores. Obviously the straight 
line is a very close approximation to the curve. The variance error 
of estimate (c?,.,) for the curve is 8.42 and for the linear approxi- 
mation 9.21 — a difference of only .79. This small difference would 
not be significant unless the group being tested numbered more than 
about 200 students. The difference between the two variances of es- 
timate is only 3.8% of the total variance of S-scores (20.83). 

Obviously, for the case used for illustrative purposes (and for 
many other cases, also), the linear estimating formula (15), or a 
slight modification of it based on (26) to take care of the propor- 
tion of questions not tried by the group, is for most practical pur- 
poses a suitable formula for scoring multiple-choice tests. Actually 
it is not a difficult formula to use; because it requires only a knowl- 
edge of the number of choices per question (k), the average group 
score (Kk), the proportion of questions not answered (q2), and the 
variance of the raw score (o’,). 

It may be observed, however, that no linear regression formula 
will change the standing of students with reference to each other. 
It will only provide an estimate of their real score. However, this 
estimate is quite important if grades for several tests are to be com- 
pared or if they are to be combined into a composite score. The old 
R — W formula for scoring true-false tests actually had the effect of 
over-weighting true-false scores when they are combined with 4- or 
5-choice questions or with nonchoice objective questions. Since many 
general tests combine objective questions of varying number of 
choices per question, it is important that the scoring formulas not 
over-weight questions of the two-choice type which contain a high 
error factor. It is also important because the use of multiple-choice 
tests frequently leads instructors to overestimate the level of student 
achievement. The bias upward in multiple-choice test scores is fre- 
quently so great as to be grossly misleading. The poor statement of 
wrong alternatives to the correct answer accentuates this problem. 
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Many questions carry alternate choices which are so easily detected 
as being wrong that few students ever check them even when they do 
not know the correct answer to a question. This avoidance of certain 
wrong answers possibly reflects an elementary knowledge but reveals 
nothing about the student’s knowledge of the subject matter of the 
course for which the test is given. Hence, the actual number of 
choices per question is most likely less than k , the intended number. 
Therefore, it is quite unlikely that the scoring formulas mentioned in 
this paper will eliminate all the upward bias in multiple-choice test 
scores. 


The Correlation Problem 


Even though the average bias in multiple-choice test scores could 
be eliminated, the error of estimate would still remain. In other 
words, the correlation between real scores and raw scores cannot be 
improved by using a scoring formula designed to eliminate the up- 
ward bias of the test scores. The use of the multiple-choice question 
which permits the student to guess at answers introduces an error 
in the score which is difficult to eliminate. Attempts to eliminate this 
error have been made by instructing students not to guess at answers, 
bolstered up by the threat to “deduct double,” or some other ratio, 
for wrong answers. Even when successful, this is an unrealistic pro- 
cedure because it not only ignores the student’s willingness to gam- 
ble, but also it ignores the fact that the student may not be absolute- 
ly sure that he does or does not know the real answer. The typical 
student is not only willing to gamble, but also he will usually prefer 
to play his intuitive hunches. 

Formulas (12) and (19) throw some light on how the choice fac- 
tor affects the correlation between real and raw scores. Formula 
(19) is of only theoretical significance because rarely will the vari- 
ance of either raw or real scores be binomial. However, formula (19) 
provides a standard for evaluating the usefulness of formula (12). 
If all questions of a multiple choice test are answered, if students 
guess at answers which they do not know, and if the variance of raw 
scores is binomial, the correlation between real and raw scores could 


—1 
(except, of course, by sam- 





not rise above the square root of 


pling error, which does not enter into these derivations. ) 
As the standard deviation of real and raw scores rises above the 
binomial standard deviation, the correlation squared can rise above 
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k-—1 
sy but if the standard deviation goes below the binomial stand- 





ard deviation, 7?,, will have an upper limit lower even than 


To illustrate, formula (12) can be written as follows: 


abd 


35 — 1 = = 
kp, Or 


(29) 

Obviously, an increase in o?, relative to np,q, will allow the limit of 

7?,, to increase; and a decrease in o”, will lower the limit of 7?,, even 
k—1 





below 


The importance and usefulness of formula (12) rests on the ex- 


tent to which o?, does exceed np,q,. Since the variance of raw scores 
can be determined empirically for any group test, a theoretical treat- 
ment of the factors and circumstances which cause o”, to be large 
hardly seems necessary. It is known, however, that o”, is dependent 
on some conditions which depress it and on other conditions which 
raise it. Most usually o”, will exceed the binomial o?, because of (1) 
a wide range of ability of students taking a test; (2) the intercorre- 
lation between replies to questions.* 

On the other hand, o”, will be depressed by (1) a narrow range 
in ability among students taking a test; (2) a high variation in ques- 
tion difficulty (see ref. 6, pages 362-3) ; and (3) a low level average 
real score which means more guessing and hence increasing impor- 
tance of the binomial factor in o’,. 

Regardless of the above considerations, the formula for r°,, is 
important and useful. Its three components, W, k, and o?, can easily 
be calculated for a given test; and the result will show the extent to 
which the choice factor has affected the reliability of the test. 


Conclusions 


1. There is an upward bias in multiple-choice test scores which is 
systematically related to the number of choices per question; i.e., 


*The editors of Psychometrika have called attention to the importance of 
this point by showing that n? enters into the formula for o,2 when even a small 
amount of intercorrelation between questions is present. (Also see ref. 6 and 
particularly its formula 19.11.) 
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—. 
R=— .. i 


This upward bias is usually greater than that defined by the 
above formula because the number of real choices per question is 
usually less than the number of intended choices. 


The usual formula for estimating real scores represents an im- 
proper reversal of the formula for estimating R from S. 
7 W 
S#R———. 
k—1 
Assuming that all students will answer all questions, the most 
suitable formula for estimating real scores from raw scores is 


~ RW—no', (ko*?,—W)R 

~ (k—1)o?, — (k—1) 0, * 
This formula is a linear equation but fits quite close to the 
curvilinear regression even though the variance of real scores 


may be much greater than that of the binomial distribution. (It 
is assumed, however, that real scores are normally distributed.) 





If real scores constituted a binomial distribution, the estimating 
formula would be 


gk R 
(k—1)R © 


If g2 questions are not answered by the group, and if the distri- 
bution of real scores is normal but not necessarily binomial, the 
regression of real scores on raw scores would be approximately 


oo ko?, —W 
— 1)o?, — W492 : 





er 


and, of course, 


A 


S=—at+d,-R. 


The scoring formulas recommended will not change the rank- 
ing of students in a class. However, if true-false questions are to 
be combined with questions of several choices or with non-choice 
questions, it is especially important that a scoring formula be 
used, Otherwise, the true-false questions will be overweighted in 
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the total test score. A correct scoring formula is also useful for 
getting a rough estimate of a group of students’ real scores — 
which are frequently much lower than the raw scores. 


7. Assuming that real scores are distributed normally but not nec- 
essarily binomially and that all students will answer all questions, 
the correlation between real scores and raw scores may be deter- 
mined approximately by 


ko?, — W 
ko?, 


2 — 
oer 
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By using the concepts of antimetry and n-chain it is possible 
to define and to investigate some properties of connectivity in a 
sociometric group. It is shown that the number of elements in a 
group, the number of antimetries, and the degree of connectivity 
must satisfy certain inequalities. Using the ideas of connectivity, a 
generalized concept of clique, called an z-clique, is introduced. 
n-cliques are shown to have a very close relationship to the existence 
of cliques in an artificial structure defined on the same set of ele- 
ments, thus permitting the determination of n-cliques by means of 
the same simple matrix procedures used to obtain the clique struc- 
tures. The presence of two or more m-cliques, where m is the num- 
ber of elements in the group, is proved to mean an almost complete 
splitting of the group. 


1. Introduction 


This paper is devoted to extending the theoretical and practical 
mathematical results presented in an earlier paper (3). In that pa- 
per it was shown that certain elementary matrix operations permit, 
to some extent, an analysis of the simple type of psychological group 
structure which is often expressed by a sociometric diagram. Spe- 
cifically, we envisage a finite set of m (> 2) elements i, 7,k, -:: 
having a structure defined on them as follows: For any two elements 
i and 7 of the set there either exists or does not exist some one type 
of “directional” relationship from i to 7. This one type of relation- 
ship on the group may assume various forms such as communication 
or friendship; and so to acquire generality in the discussion we shall 
speak of an antimetry from i to 7. Then communication is merely 
a special type of antimetry. It is arbitrarily assumed that no anti- 
metry can exist from an element 7 to itself. By “directional” is 
meant that knowing of the existence or non-existence of an anti- 
metry from i to j does not give any information about the existence 
or non-existence of an antimetry from j toi. For example, if the 
form of the antimetry is “i communicates to j,” then the specific 
knowledge that a can communicate to b does not tell us, in general, 
whether b can communicate to a. 
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To such a structure of antimetries we associate a matrix 
G = [gi;] as follows: If there exists an antimetry from 7 to 7 then 
write the number 1 in the gi; entry of the matrix, and if no such 
antimetry exists place a 0 in the entry. Since we assume no anti- 
metry can exist from i to i, the main diagonal will always be 0’s. 
If there is both an antimetry from i to 7 and from 7 to 7, then we 
say a symmetry exists between i and 7. From the matrix G we may 
extract a symmetric matrix S, called the matrix of symmetries, as 
follows: If a symmetry exists between 7 and j then si; = 8;; = 1 
and otherwise s;; = s;;=0. 

In (3) there are two theoretical results which will be utilized 
here and which, for convenience, we shall briefly summarize. How- 
ever, where the method of determining cliques is necessary to apply 
the results of Sections 2, 3, and 6, it will be assumed that the reader 
is familiar with the procedure as outlined in that paper. 

Let us define an u-chain from i to 7 as an ordered set of n+1 
elements, with 7 the first and 7 the last, such that there is an anti- 
metry from i to the second element, an antimetry from the second 
to the third, etc. and an antimetry from the n™ element to 7. If the 
antimetry is communication then an n-chain is a path of indirect 
communication from i to 7 in steps. Two n-chains from 7 to 7 will 
be considered to be distinct if the k‘" element of the first chain dif- 
fers from the k*’ element of the second for some value of k between 
2 and n. It was shown that a positive integer c occurs in the 77 
entry of the n™ power of G, i.e., gi; —c, if and only if there are 
c distinct n-chains from i to 7. 

An interpretation of this result in terms of the communication 
of information from element to element may give some insight into 
this formalism. We can imagine that the structure matrix G repre- 
sents a linear transformation of a vector of information in the fol- 
lowing way: Let a; be the symbolic representation of the informa- 
tion introduced into the group at element 7, then the row vector 
A = [a@,, Q@,-++, Gm] represents all the information introduced into 
the group. If we suppose that each element transmits all the infor- 
mation it has to each other element to which it is connected by an 


m 
antimetry, then element j will receive the information 5 aigi;. In 


$=1 
order to permit a matrix interpretation of this sum, addition will 
have to follow the law k + k = 2k, in which case the operation in- 
dicated is the matrix product AG, which is a row vector represent- 
ing the distribution of information in the group after one inter- 
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change of information. Because of this law of addition, the distribu- 
tion after n interchanges will be (AG"")G = AG". 

It may be helpful to discuss this law of addition and one other 
equally plausible law. The information a; may be considered to be 
composed of members of a basic set of elementary particles of infor- 
mation (statements, for example). Then, in general, a; and a; will 
have some common members, i.e., element i will have some of the 
information that element j has. If 7 transmits all its information to 
i, then, in a certain sense, 7 will only gain that which it did not al- 
ready have. Therefore, addition will be set theoretical: k + k= k. 
This law requires that Boolean matrices (1, 213; 2, 184-195), rather 
than ordinary ones, be used if the resulting algebra is to be associa- 
tive.* On the other hand, the law k + k = 2k is useful because it 
indicates how often the same item of information will recur at any 
point in the group. Frequently the recurrence of an item is at least 
as significant as its existence. But, for our purposes, the importance 
of this interpretation is that it leads to the simple manipulations of 
matrix algebra. 

Now, let us define a clique C to be a subset of not less than three 
elements of the original m elements which has the property that be- 
tween every pair of elements of C there is a symmetry, and, in addi- 
tion, C is not a proper subset of any other set of elements for which 
the first property is true. (A is a proper subset of B if and only if 
every element of A is an element of B and there is at least one ele- 
ment of B which is not an element of A). This requires that a clique 
is not a proper subset of a clique; however, two or more cliques may 
overlap in the sense that they have one or more elements in common. 
It can be shown that: “An element 7 is contained in a clique if and 
only if in the matrix S* the main diagonal entry corresponding to i 
is positive” (3, §5, Theorem 3). This theorem greatly simplifies the 
analysis of a structure, since such tightly knit subsets are frequently 
considered to be the basic parts of the structure. 

One can object to this definition, however, on the basis that, 
cliques being maximally connected sets of elements, the analysis 
omits substructures which are somewhat less restrictive, but which, 
for all practical purposes, would be called cliques. This difficulty was 
pointed out and a generalized definition was suggested which would 
select those ordered sets of n elements which are connected by a sym- 
metry from the first to the second element, the second to the third, 

*The present author has conducted some investigations on the elementary 


theory of Boolean matrices and their application to sociometrie group structures. 
This work may be published in the future. 
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etc., the ‘" to the first (3, §2.10). These structures, when drawn, 
appear as “circles” of symmetries. Such a definition is not useful, 
since an analysis in terms of it depends on the unsolved problem of 
redundancies and, moreover, does not take into consideration more 
complex clique-like structures. This latter difficulty seemed insur- 
mountable; in fact, it was stated: “The most general definition of a 
clique-like structure including antimetries will not be discussed, for 
it is believed that this will not be amenable to a concise mathemati- 
cal formulation.” This is now known to be incorrect. We shall pre- 
sent a definition of a generalized clique, called an n-clique, which is 
readily evaluated by means of matrices through the determination 
of the cliques within a psuedo-structure. This definition is more val- 
uable than the earlier definition since, first, permitting antimetries 
and more complex connections by symmetries, it is more general; 
and, second, the analysis using this definition is independent of the 
problem of redundant n-chains. 

The study of this particular problem depends on a concept of 
connectivity in the entire structure. But, since the study of connec- 
tivity in a structure leads to results that are intrinsically useful, we 
may divide this study into two parts: 1) connectivity in a structure, 
§2; and 2) a generalized definition of clique §3. 


2. Connectivity in a Structure 


In this discussion we shall be concerned only with the number 
of elements on which a structure is defined and the number of anti- 
metries in that structure. We may denote any structure of p anti- 
metries defined on a set M of m elements by the symbol G(m,p). We 
define G(m,p) to be properly n-connected if for every pair of ele- 
ments 7 and 7 (not excluding the possibility that 7 = 7) in the set M 
there exists at least one q-chain from i to 7, with q some positive in- 
teger no larger than ”. This is to say, if G(m,p) is a properly 1-con- 
nected communication structure, any element 1 may communicate to 
any other element j and itself, in n steps or fewer. Since any n-chain 
encompassing more than m elements must be redundant in the sense 
that at least one element occurs more than once in the chain, it is 
clear that if the structure is not properly m-connected it never will 
be properly connected. Since we assumed earlier that there is never 
an antimetry from an element to itself, these two statements show 
that the range of n is all integers from 2 through m. 

It is not difficult to see that if a structure G(m,p) is properly 
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n-connected, and if G is the matrix representation of the structure, 
then the sum 


G+ Gt+-.+G@ 


will contain no zeros as entries (see §6). In general the connectivity 
of the structure must be determined in this way, but in addition it is 
useful to remember that if any row or any column of G contains ail 
zeros then G(m,p) is not properly connected for any n (§6, Lemma 
1). We see as an immediate consequence of this that in a properly 
connected structure the number of antimetries, p, must be greater 
than or equal to the number of elements, m. It can also be shown 
that if » =m and the structure G(m,m) is properly n-connected, then 
the degree of connectivity, n, must be m (§6, Lemma 3). The upper 
bound on » is easily seen to be m(m—1) by considering the matrix 
representation of a structure and recalling that the main diagonal 
must be all zeros. 

The several statements in the last two paragraphs permit us to 
write: If G(m,p) is a properly n-connected structure, then 


2<n<m (1) 
m<p<m(m—l1). (2) 


Further, it is conjectured that if G(m,p) is properly n-connected, 
then 


Qn<n+p, (3) 


but this has not yet been proved. An examination of the results ob- 
tained (§6) and an attempt to construct a counter-example will soon 
convince the reader that this relation is probably true. 

It can be proved, however, that for any positive integers satis- 
fying these three inequalities there exists at least one structure 
G(m,p) which is properly n-connected (§6, Theorem 1). This must 
not, however, be taken to mean that such a structure may not be 
properly qg-connected for some gq less than n, i.e., it need not be 
“exactly” n-connected. For example, if » = m(m—1) then G(m,p) 
is necessarily properly 2-connected, so if n is chosen larger than 2 
it is impossible to find a structure G(m,p) with p = m(m—1) which 
is exactly n-connected. This question of a bound on p such that there 
exist structures which are exactly n-connected will be taken up in 
. greater detail after first considering the special case which exists 
when 2m =n + p. 

It is particularly interesting to know which are the structures 
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for which the equality 
2n=n+p (4) 


is satisfied, since, if the conjecture, eq. (3), is true, these are mini- 
mal structures in the sense that for a given number of antimetries 
we obtain the minimum proper connectivity possible; or, conversely, 
for a given proper connectivity, x, the minimum number of anti- 
metries required to produce such connectivity. It can be proved 
(§6, Theorem 1) that when the quantities m,n”, and p satisfy equa- 
tions (1) and (2) there are structures for which the equality (4) 
does hold. In that existence proof, such a structure is exhib- 
ited: Select any » of the m elements and form a single n-chain 
from one element back to itself using all ~ elements in the chain. 
This structure, which is denoted by R(m), appears when sketched 
as a ring, with the antimetries head to tail. Connect by symmetries 
each of the remaining m—n elements to some one element of 
R(m). The m—n symmetries exist, for by equation (4) we have 
p—n = 2(m—n) antimetries remaining after R(m) is formed. In 
general, such minimal structures are not unique even up to a renum- 
bering of the elements. 

This minimal equation and the associated minimal structures 
may prove to be of some interest in designing communication net- 
works which are to contain the minimum number of connections to 
effect a certain degree of proper connectivity. Such a specification 
on the proper connectivity is often imposed in practical problems by 
considerations of efficiency. In general, however, it is to be expected 
that the faults of the rather pathological structures presented will 
out-weigh their minimal nature. For example, the element in R(m) 
symmetrically connected with the m—n elements outside R(m) will 
probably tend to be overloaded with messages in any practical case. 
It may be, however, that there are less objectionable minimal struc- 
tures, so an interesting problem is to find a simple general statement 
of all possible minimal structures. 

Let us again define a structure G(m,p) to be exactly n-connected 
if it is properly n-connected but not properly g-connected for any q 
less than x. It was stated earlier that for integers satisfying equa- 
tions (1), (2), and- (3) it is always possible to find a structure 
G(m,p) which is properly n-connected, but that it is not necessarily 
true for all such m, 1, and p that there exist structures which are 
exactly n-connected. The significant inequalities for exactly n-con- 
nected structures are: 


2<n<m (5) 
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p < (m—n—1) (m+2) + n(n +8) /2 (6) 
n=2 (5a) 

p <m(m—1) (6a) 

n=m (5b) 

p<m+ (m—1) (m—2)/2. (6b) 


The following two theorems have been proved: If a structure 
G(m,p) is exactly n-connected, then for the correct equation (5), 
(5a), or (5b), determined by the value of n, the corresponding 
equation (6) is satisfied (§6, Theorem 2). It can further be shown 
(§6, Theorem 3) that given integers m,n, and p satisfying any one 
of the following three-sets of equations, which are a combination of 
equations (3) and (6) for each equation (5), there exists at least 
one structure G(m,p) which is exactly n-conditioned: 


2<n<m (7) 

2m—n <p < (m—n—1) (mM+2) + n(n+8) /2 (8) 
n=2 (7a) 

2(m—1) < p < m(m—1) (8a) 

n=™m (7b) 

m<p<m+ (m—1) (m—2) /2. (8b) 


In the proofs of Theorem 3 and Lemma 10 an example of at least 
one such structure is presented for each admissible set of m, n, 
and p. 

This work suggests that it might be possible to carry out a sys- 
tematic classification of all exactly n-connected structures in terms 
of the parameters m,n, and p. Such a classification, were it sim- 
ple, would certainly prove useful in the design of group experiments 
to test the importance of certain structural parameters. 


3. A Generalized Definition of Clique 


We shall say that a subset C of the set M is n-connected if for 
every pair of elements 7 and j in C, with i # j, there exists at least 
one q-chain in G(m,p) from i toj where q is not larger than n. We 
note that this definition permits the chains which produce the con- 
nectivity of C to contain elements outside C and, furthermore, does 
not require any element be connected to itself. Thus for n-connected 
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subsets,1 < n < m. Had the earlier definition of “properly n-con- 
nected” been applied to C as though C were the entire set, then all 
q-chains would have to remain in the set C and there would have to 
be a q-chain from i to i for any iin C. It is only to this extent that 
the two definitions differ, but this difference is significant both in the 
generality acquired (since the present definition includes the for- 
mer) and in a systematic correlation with matrix procedures. 

A subset C of M which consists of three or more elements is said 
to be an n-clique if and only if C is an n-connected subset in the 
structure G(m,p) which is not a proper subset of an n-connected 
subset of M. As in the definition of a clique, a proper subset of an 
n-clique is not called an n-clique, but this does not obviate the pos- 
sibility that n-cliques may overlap. 

The following statements are immediate consequences of the 
definition: If C is an n-clique, then C is a subset (not necessarily 
proper) of a q-clique for all g > ». Every clique is a subset of an 
n-clique for all nm, 1 < n < m; however, the converse, that every 
n-clique contains a clique, is not true. The cliques and 1-cliques of 
a structure are identical. These last two statements justify the con- 
tention that an n-clique is a generalized concept of a clique. If the 
entire structure G(m,p) is properly n-connected, then evidently the 
set M is an n-clique. 

From a consideration of this definition it seems probable that, 
other than cliques, only 2- and 3-cliques will be of any practical 
value, since any other n-cliques are very loosely connected. How- 
ever, the 2- and 3-cliques, without possessing the exacting restric- 
tions of a clique, are still sufficiently restrictive to be of general 
value in analyzing the structure. Happily, only these low values of 
nm are of interest, since this keeps to a minimum the work required 
in the following matrix analysis for n-cliques. 

To determine the n-cliques of a structure G(m,p), written as a 
matrix G, we first compute the following sum: 


A(n) = [wJ=G+Gr+--+Gr=| 3 9 


q=1 
From this we obtain the pseudo-structure B(n) on the set M de- 
fined by the matrix B = [b;;] where 
bii=0 foralliinM; 
b:;=1 if aj;>od, t# 4; 
6:;;=0 if aij—0. 
Let S be the matrix of symmetries extracted from B (see §1). From 
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the cube of this matrix S we obtain the cliques of the pseudo-struc- 
ture using the theorem on cliques stated in the introduction or in §6. 
For further details on this procedure see (3), §§3, 4. It has been 
shown (§6, Theorem 4), that the n-cliques of G are identical with 
the cliques of the pseudo-structure B(n). This theorem gives a pro- 
cedure to determine n-cliques which is dependent only upon our abil- 
ity to determine cliques and which requires, for n = 2 or 3, no more 
than twice the matrix manipulation required to solve for cliques. 

Had the definition of an n-connected subset required the exist- 
ence of a q-chain, gq < n, from an element to itself as well as from 
any element to any other element of the subset, then the theorem 
would have been: A subset C of M is an n-clique of the structure 
G(m,p) if and only if (a) C is the largest subset of a clique C’ of 
B(n) for which a;; > 0 where 7 is contained in C’, and (b) C con- 
sists of three or more elements. That is to say, if X is the subset 
of elements of M for which A(n) has a non-zero main diagonal en- 
try, and Y is a clique of the pseudo-structure B(n), then those ele- 
ments common to X and Y, provided there are three or more, form 
an n-clique; and every n-clique is formed by such an intersection. 

In general, if these different definitions are applied to the same 
structure the results obtained will differ. In selecting one in favor 
of the other the following two facts must be kept in mind: (a) the 
determination of »-cliques using the second definition is slightly 
more difficult, but (b) the second definition is more restrictive and 
so will select more tightly knit structures. Sometimes it may be that 
this additional restriction is sufficiently desirable to offset the added 
complication in evaluation. 

About m-cliques a further statement can be made (§6, Theorem 
5). If in a structure on m elements there exist two or more m-cliques, 
then as sets of elements they are disjoint. In other words, if i is a 
member of one m-clique it is not a member of any other m-clique. 
More generally, if C and C’ are two m-cliques, then, in the original 
structure G(m,p), there can only be n-chains (1 < » < m) from 
the elements of C to those of C’ or vice-versa; never both from C to 
C’ and C’ to C. In particular, there may not be a symmetry between 
an element of C and one of C’. This last result may be of some in- 
terest in practical communication problems, for the existence of more 
than one m-clique indicates that the structure seriously divides the 
set of elements into disjoint subsets completely incapable of effec- 
tive intercommunication. 
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From this sum we may immediately write down the matrix of sym- 
metries of B(2). 





182 £6878 9 
1, eee ei Oe S11 
ei 66.8 ts £ 4 6 6 0 
31-28 @ 228 8 6 1 14 
15243434 63 82 * +.2 

S=5/1011000001 
¢'+@1060¢060 86 © 6 1 6 
710100010 0 0 0 
gi 00601006001 0 
er Er@ Let ote @ 
10 PPL EL 6G eo O | 





Squaring and then cubing S , we obtain as the main diagonal 


it £3 €@€ © 6 T&F W 
MW2wewwe 2a 2s OE. 


Since the elements 1,3, 4,5, and 10 all have numbers as large 
as 12 as main diagonal entries, we expect them to form a clique 
of five members. The 16 associated with elements 1 and 3 suggest 
they are in another clique, and the 18 of element 4 suggests it is in 
one or two other cliques. After eliminating elements 2, 6, and 7 as 
a clique not overlapping any others, we find that 4, 8, and 9; and 
1,3,4, and 9 are two cliques. It is easy to verify that these four 
cliques and their regions of overlap account for the values on the 
main diagonal. In summary, the 2-cliques of G, i.e., the cliques of 
B(2), are: (1,3,4,5,10), (1,3,4,9), (4,8,9), and (2,6,7). 

This example illustrates the fact that a clique is a subset of an 
n-clique, since the clique of G is (1,3,4), and also the fact that there 
may be n-cliques which are not merely extensions of cliques, e.g., the 
two different 2-cliques of three members each. 


5. A Remark 


Professor Alex Bavelas, of the Massachusetts Institute of Tech- 
nology, has mentioned to the author that various experimenters 
have, in certain cases, observed that the introduction of a stress sit- 
uation into a fairly large group of individuals (say, 12 or more) pro- 
duces a marked “splitting” of the group. For example, suppose that 
a group of children have been together for such a period that friend- 
ship patterns have been established and that the group regularly has 
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occasion to use certain tools. We shall imagine that there are enough 
tools so that normally there is little or no friction caused when they 
are to be selected for use. Now suppose that one day the number of 
tools available is markedly decreased, say, totwo or three. It is not un- 
reasonable to expect, and it has been observed to occur, that in some 
cases this will cause the group to break into two or three subgroups 
which will be associated with the tools available. It is certainly an 
interesting theoretical problem to understand, and to predict if pos- 
sible, the manner in which the group splits under such a stress. Is 
the knowledge of a significant structure on the group (friendship in 
this case) sufficient to predict, within a certain error, the manner 
in which the group subdivides? It is conjectured that there may 
well be a strong correlation between the sets into which the group 
divides when under stress and the existence in such a structure of 
relatively large 2- or 3-cliques having little or no region of overlap. 
By relatively large we mean of the order of one-third to two-thirds 
of the members of the original group. Cliques probably would not 
be sufficiently general to make this correlation, for they do not asso- 
ciate together some elements of the set which we think intuitively 
are actually quite closely connected in the structure. But, as the 
above example has shown, 2-cliques certainly bring out more bonds 
in the structure than do cliques; so at this level, or possibly on the 
level of 3-cliques, the correlation may be possible. The carrying out 
of this type, and more abstract types, of experiment will prove usefui 
in selecting and modifying the mathematical definitions introduced 
in any theory pertaining to group psychology. 


6. Mathematical Theory 


Let M be a finite set of m (m > 2) elements a, b,c,-:--, 
7,3, h, +--+ on which there are defined p antimetries. A statement 
of the p antimetries defined on the set M is said to be a structure 
and is denoted by G(m,p). The structure will have the matrix reali- 
zation G = [gi;] , and thus we may denote an antimetry from i to 
jasgij=1. 


Definition 1: A structure G(m,p) is properly n-connected if and only 
if for every pair of elements 7 and 7 contained in M there exists a 
q-chain from i to j with 1 < q <n. 

Since, by the definition of an n-chain (3, §5.03), any n-chain 
for which n > m is redundant, and by assumption no antimetry ex- 
ists between an element and itself, the degree of proper connectivity, 
n, satisfies the condition 2< n < m. 
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G(m,p) is properly n-connected if and only if > G2 > 0, where 
0 is the zero matrix. “i 
Proof: If G(m,p) is properly n-connected, there exists a q-chain, 
1<q<7n, from i to j for all 7 and j contained in M; i.e., gi; =1, 
so > gi; > 1 > O since there are no negative contributions to the 


q=1 
sum. 


If } g? > 0, then for every 7 and j there exists a q such that 


ql 
gi; =1> 0. Then by Theorem 1 of (3) there exists a g-chain 
from i to j with 1l< q<n. 
Lemma 1: If a row or a column of G is empty, then G(m,p) is not 
properly n-connected for any n. 


Proof: Suppose that row i is empty, then it is impossible for an 
n-chain to exist from 7 to any element 7 for gi; = gixgxj° = 0, 
since gix = 0 for all k. A similar argument holds for an empty col- 
umn. 


Corollary: If G(m,p) is properly n-connected, then 
m<p<m(m—1). 


Proof: 1. m < p. Suppose py < m, then some row is empty, for 
there is a total of m rows in the matrix. By Lemma 1, G(m,p) is not 
properly n-connected for any n. This is contrary to hypothesis, so 
pezm. 


2. pw < m(m—1). Consider the matrix representation of the 
structure. By definition gi; = 0 for the m 7’s of M and there are m? 
possible entries in an m by m matrix, so at most p = m?— m. 


The G(m,p) structure defined by the matrix G: 
Jin —1 (Ift—m, then i+ 1=1.) 
9i; =O forall other] 

is denoted by R(m). 


Definition 2: In a structure G(m,p) there exists an exact n-chain 
from 7 to 7 if and only if there exists an n-chain from i to j but no 
qg-chain, gq < n, fromito7. An exact n-chain from i to j is denoted 
by E'(i,j) and the set of elements included in the chain is called E. 
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Definition 3: A structure G(m,p) is exactly n-connected if and only 
if it is properly n-connected and there exists at least one exact 


n-chain in the structure. 


Lemma 2: There exists at least one structure G(m,m), specifically 
R(m), which is exactly m-connected. This is easily verified. Ob- 
serve that between any two elements there is only one n-chain in 
Rim). 


Lemma 3: If G(m,m) is properly n-connected, then it is exactly 
m-connected. 


Proof: By Lemma 2 there are structures for which the hypothesis 
holds. By the argument following Definition 1, n < m. 


Suppose x < m. By Definition 1 there exists, for any 7 con- 
tained in M, at least one q-chain (2 < q < n < m) fromi tot: 
Gi” = Gij -*+ Oxi = 1. Since n < m, there exists some element a 
not included in this g-chain; but, by hypothesis, there is an r-chain 
(2<r<n<m) fromitoa. Beginning at 7 these two chains may 
have some antimetries in common, but there is a first element b (pos- 
sibly b = 7) after which they will differ, since, if they did not, a 
would be included in the original g-chain. Thus the two chains may 
be written: 

Diz YoYear** Jxi — 1 . 
Jijere DieDes se He 


Thus Joc = Joe — 1 andc # e. Thus row D has two antimetries, but 
there are only m antimetries in the structure by hypothesis, so that 
some row must be empty. Lemma 1 then implies that G(m,m) is 
not properly n-connected fer any x < m, which is contrary to hy- 
pothesis, son =m. 


It is felt that by the proper use of this lemma it should be pos- 
sible to prove the following theorem: If G@(m,p) is properly n-con- 
nected, then 2m < n + p. This has not been done. 

Theorem 1: For any integers m,n, and p satisfying 2 < n < m and 
2m = n + p, there exists at least one structure which is properly 
n-connected. 


Proof: Consider the structure P(n) defined as follows: On a set R 
consisting of n of the elements of M form the structure R(n) using 
n antimetries. Select one element a of R and connect each of the 
remaining m—mn elements to a by a symmetry. This is possible be- 
cause there are p — n = 2(m—n) antimetries, or m—n symmetries, 
remaining. 
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P(n) is properly n-connected: In general there are four classes 
of q-chains from any element 7 to any element j: 


1. If iand j are contained in R, then, by Lemma 2, there is a 
q-chain, 1 < q < n, fromitoj. 

2. If 7 is contained in M—R and j is contained in R, then 
either 

a. j =a S8o that gi; = gia = 1 by construction, or 


b. jg #a8o that giagai? = gij* =—1 with 1 < q < n—1 since 
the only n-chains in R(n) are from an element to itself. 


3. If zis contained in R and j is contained in M—R, the argu- 
ment is the same as in 2. 


4. If iand 7 are both contained in M—R, then, by definition of 
@, Gi; = GicQaj — 1 and 2< n by definition. 


Corollary: For any integers m, n, and p satisfying 2 < n < m, 
m <p < m(m—1), and 2m < n+ p there exists a structure G(m,p) 
which is properly n-connected. (This structure is not in general ex- 


actly n-connected). 


Proof: Let p’ = 2m — n. Since 2m < n+ p it follows that p — p’ > 
0. Sincee2 <n<m,m < 2m—n=p < 2(m—1)< m(m—1). By 
Theorem 1, there exists a structure G’(m,p’) which is properly 
n-connected. To this add py — p’ antimetries in any arbitrary fash- 
ion. This forms a structure G(m,p) which is properly n-connected, 
for the addition of antimetries can do nothing but decrease the value 
of n required for proper connectivity. 


Lemma 4: If G(m,p) is exactly m-connected, then there exists an 
element a such that for some b andcinM,b#a#c,gu=—Ja=—1, 
but g1i = 0 fori# band gj;.—Oforj#c. 

Proof: By Definition 3 and the definition of an n-chain there exists 
an exact m-chain E(a,a) for some element a to itself. Clearly there 
exists b = a + 1 and c = a — 1 such that gaan = Yoia —1 since 
E(a,a) is an n-chain. 

Suppose there exist an 7 and/or a j such that 1 # a + 1 and 
9a — 1 and j #a—1 and gj =—1. (Such i and j will not exist if 
m = 2, but then the lemma is trivial). Without loss of generality, 
7 may be assumed to exist. Then the m-chain 


E(@,@): QanGar,a+2 *** Jota — 1 
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is not exact since 
GaiQi,i+1 sas Qa-1,a — 1 


is a g-chain froma toa and g < m—1,sincei#aandi#atl. 
A similar argument applies to the existence of a 7. Thus E'(a,a) is 
not an exact m-chain, which is contrary to choice, so the lemma is 
true. 


Lemma 5: If G(m,p) is exactly m-connected, then 
p<m-+ (m—1) (m—2)/2. 


Proof: 1. The lemma is true for m = 2, for p =n =m = 2, So 
p=2< m+ (m—1) (m—2)/2=2. 


2. If the lemma is true for m elements then it is true for 
m-+1. For suppose that it is not true for m + 1 elements, i.e., there 
exists a G(m+1,p) which is exactly (m+1)-connected and p > m + 
1 + m(m—1)/2. By Lemma 4 there exists an element a = m + 1 
(renumbering if necessary) for which gm: = Ym, = 1 are the 
only antimetries involving m + 1. Remove this element and these 
two antimetries, replacing them by gm, — 1 if it is not already in 
the structure. It is possible that gm; — 1 for 2 < i < m—1. Remove 
these antimetries if they are present. This removes in total no more 
than 2 + m—2 —™m antimetries from the structure, leaving a struc- 
ture G(m,p’) with 


p>p—m>1+m(m—1)/2=—m + (m—1) (m—2) /2. 


G(m,p’) is exactly m-connected: It is properly m-connected by 
a chain from m to m since there was a chain (exact) from m + 1 
tom + 1in G(m+1,p). It is exactly m-connected, for suppose there 
is a g-chain 9mi9i,m° =1,q < m, then in G(m+1,p) the (q+1)- 
Chain 9ms1191,m'%) Jmms1 = 1, q +1 < m+ 1, would exist, which is 
contrary to assumption. This considers all possibilities, since gm. = 1 
was the only antimetry added, and g»; = 0 except fori=1. 

Thus G(m,p’) is exactly m-connected and 


p >m+ (m—1) (m—2)/2, 


which is contrary to the induction assumption, so the lemma is true 
for m+1 elements. Thus the lemma is true in general. 

Lemma 6: If G(m+1,p) is exactly m-connected and there exists an 
exact m-chain E'(a,b),a # b, then p < m(m+8)/2. 

Proof: By renumbering the elements, let a = 1 and b = m + 1 and 
all other elements of the set have the number of their position in 
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E(1, m + 1). The antimetries 9; = 1,1 +2 < j < m+ 1 and 
1 <i < m—1 may not occur in G(m+1,p) ; for if they could, then 
9i; = 1 could replace any gi; =—1,q > 2, occurring in the struc- 
ture. Thus the m-chain E(1,m+1): Qi29o3 -** Gij ++ Inman = 1 
could be replaced by gi2923 °** Yij *** Ymm+1 —= 1 which is an r-chain, 
r=m—q+i1<™m. This is contrary to the hypothesis that 
E(1,m+1) is an exact m-chain; hence these antimetries cannot exist. 
These are (m—1) + (m—2) + --- + 1 = m(m—1)/2 antimetries 
which may not be present. At most there are m(m+1) antimetries 
present in any structure on m+1 elements, by the corollary to Lemma 
1, sop < m(m+1) —m(m—1) /2 = m(m+3)/2. 


Lemma 7: If E' (1,7) is an exact m-chain from i to 7,7 #7, on a set 
E of n+1 elements and k is an element not in E', then between k 
and the elements of £ there are at most n+4 antimetries. 


Proof: Suppose the lemma false, that is, that there are more than 
n+4 antimetries. Then there are at least four elements of HE, say 
a,at+l1,a+2, and a+3 such that there are symmetries between & 
and these four elements, since the set E consists of +1 elements. 
We may suppose that the elements of E have been renumbered ac- 
cording to the order of their occurrence in E(i,7). Then Qa,a+19a+1,0+2 
Ya+2,0+3 — 1 is a q-chain, gq > 3, which may be replaced by ga9x,013 = 1, 
so there exists an r-chain from 7 to 7 with r=n—q+2<n. This 
is contrary to assumption, and the lemma is true. 


Lemma 8: If E(i,i) is an exact n-chain on a set E of n elements and 
k is an element not in E , then between k and the elements of E there 
are at most n+2 antimetries. 


Proof: A slight rewording of Lemma 7 suffices to prove Lemma 8. 
Theorem 2: If G(m,p) is exactly n-connected, 2 < n < m, then 
p< (m—n—1) (m+2) + n(n+8)/2. 

Proof: By Definition 3 there is at least one exact n-chain. Since 
n <m, either it is of the form: 


1. E(i,j),i #7. The elements of E can have at most n(n+38) /2 
antimetries on them (Lemma 6). 

The m—n—1 elements of M—E can have at most (m—n—1) 
(m—n—2) antimetries on them (Corollary, Lemma 1). 

Any element of M—E can have at most +4 antimetries with 
the elements of E and there are m—n—1 elements in M—E,, so this 
is at most (m—n—1) (n+4) antimetries (Lemma 7). 
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Since this enumeration includes all possible antimetries in an 
exactly n-connected structure, their sum is the maximum possible 
number of antimetries where there is an exact n-chain on +1 ele- 


ments: 
p< (m—n—1) (m+2) + n(n+8) /2. 


Or 


2. E(i,i). The elements of / can have at most n + (n—1) 
(n—2) /2 antimetries on them (Lemma 5). 


The m—” elements of M—E can have at most (m—n) (m—n—1) 
antimetries on them (Corollary, Lemma 1). 

Each element of M—E can have at most +2 antimetries with 
elements of E', and there are m—n elements in M—E. This is at 
most (m—n) (n+2) antimetries (Lemma 8). 


As in part 1., add these numbers: 


p< (m—n) (n+2) + (m—n) (m—n—1) + 2+ (n—1) (n—2) /2 
= (m—n—1) (m+2) + n(n +8) /2—n+38 
< (m—n—1) (M42) +n(n+4+8)/2 if n28. 


Lemma 9: If m and p are integers such that 
2<m<p <m+ (m—1) (m—2)/2, 


there exists a structure G(m,p) which is exactly m-connected. 
Proof: Consider the structure R(m) on the m elements. This, by 
definition, contains an exact m-chain E(1,1) from element 1 to it- 
self. Renumber the elements of M according to their order of oc- 
currence in #(1,1). For each element 7, 3 < 7 < m, introduce the 
antimetries gi i —1,k=—1,2,---,i—2. This is a total of 


S (2) =Si= (m1) (m—2)/2 


antimetries added to the original m. This structure we shall denote 
by Q(m). 

It is necessary to verify that each such structure is exactly 
m-connected. When p = m, Q(m) = R(m); and Lemma 2 proves 
that it is exactly m-connected. The exact m-chain E(1,1) can only 
be removed, that is no longer be exact, by the introduction of anti- 
metries of the form: 

l. gi =—1,722#m. These are not introduced because, for any 
1,1—-k >i— (t-2) =2>1. 
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2. gij =1,7 >tiandj7 #1. These do not occur for any 7 be- 
cause i > i—k = jsincek 71. 


Lemma 10: If p = m(m—n+1) + (n—1) (n—2)/2 and2 <n <m, 
then there exists a G(m,p) which is exactly n-connected. 

Proof: 1. If n=2 the equation reduces to p = m(m—1) which, by 
the corollary to Lemma 1, is the maximum possible number of anti- 
metries. This structure is obviously properly 2-connected, and from 
definitions 2 and 3 it is easily seen to be exactly 2-connected. 


2. If n= the equation reduces to p = m(m—8+1) +1 = m — 
2m + 1. Connect maximally the set M — {a} where a is contained in 
M. This is (m—1) (m—2) antimetries on M — {a}. Let b andc, 
b #c, be elements contained in M — {a}; such elements exist be- 
cause m > n= 38. Introduce g.. — 1 and ga —1. Also introduce 
the antimetries gi, — 1 where 7 is contained in the set M—R = M — 
{a} — {b} — {c} of m—3 elements. These introduce exact 3-chains 
from a to @: Qav9viJia — 1 is the shortest chain from a to a, since 
Jai = 0, 7% contained in M—R, gn. = 0, and gue = 0 by construction. 
In total (m—1) (m—2) + 2+ m— 3 = m? — 2m + 1 antimetries 
are introduced. 


3. n>3. Onaset R of n of the elements of M form the struc- 
ture Q(m) of Lemma 9 having n + (#—1) (n—2)/2 antimetries. 
Number the elements of the exact n-chain from 1 through n as in 
Lemma 9. On the set M—R + {n} + {n—1} + {n—2} of m—n+3 
elements introduce all possible antimetries except Qn-o. = 1, i.e, a 
total of (m—n+3) (m—n+2) — 1 antimetries. However, this in- 
cludes five antimetries already considered in Q(m), specifically: 
Inn-1 = hs YJn,n-2 = ee Yn-1,n-2 = 1, Qn-2n—1 = a Ie and Guin — 1. From 
the m—n elements of M—R introduce antimetries to the n—4 ele- 
ments 2,3,---,n—3 of R. This gives a total of n + (n—2) (n—1) /2 
+ (m—n+38) (m—n4+2) —1—5 + (m—n) (n—4) = m(m—n4+1) + 
(n—1) (n—2) /2 antimetries. 


The verification that this structure is exactly n-connected is 
straightforward but lengthy, so it will be omitted. 
Corollary: If m, n, and p are integers such that 2m — n < p < 
m(m—n+1) + (n—1) (n—2)/2 and 2 < n < m, then there exists a 
structure G(m,p) which is exactly n-connected. 


Proof: The construction of Theorem I gives a structure which is ex- 
actly n-connected for p = 2m—n. The construction of Lemma 10 
adds, but does not remove, antimetries to this structure to produce 
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one which is exactly n-connected for 
p=m(m—n+1) + (n—1) (n—2)/2. 


Thus any value of p between these two bounds will permit the con- 
struction of an exactly n-connected structure. 


Theorem 3: If m,n, and p are integers such that for 2 <n<™m, 
2m—n < p < (m—n—1) (m+2) + n(nt+8)/2; for n = 2, 2(m—1) 
< p < m(m—1); and forn=m,m < p < m+ (m—1) (m—2) /2; 
then there exists a G(m,p) which is exactly n-connected. 

Proof: The existence of exactly n-connected G(m,p) for 2 <n < m 
and 2m— < p < m(m—n+1) + (n—1) (n—2) /2 is proved in Lem- 
ma 10. This covers the case of n = 2 and n = m stated in this theo- 
rem. Thus we need only consider 


m(m—n+1) + (n—1) (n—2)/2 < p < (m—n—1)(mt+2) + 
n(n+8) /2 for 2 << m. These limits on p differ by the number 
n—3 which is non-negative for n < 38. 


It can be verified that the following structure G(m,p) is exactly 
n-connected for p = (m—n—1)(m+2) + n(n+38)/2: An exact 
n-chain E (a,b), a # b, is connected maximally as in Lemma 6 and 
the set M—E is connected maximally as in Lemma 1. Place 
Cis — 1 if 7 is in M—E and 7 isin E »POi—1, 943 1, and 724 — 1 
if 7 is any element in M—E. Let all other gi; = 0. 


It is easily verified that n(n+1) /2 of the antimetries introduced 
on F are inessential to E'(a,b)’s being an exact n-chain and to the 
structure’s being connected. Since n(n+1)/2 > n—3 for n > 3, any 
number of these antimetries up to and including n—3 may be re- 
moved. On verification that each of these structures is exactly 
n-connected, the theorem is proved. 

Definition 4: A subject C of M is n-connected if and only if for every 
a and 7, 1 # 7, contained in C there exists at least one q-chain, 
1<q<n,inG(m,p) fromitoj. 

Definition 5: A subset C of M having three or more elements is an 
n-clique, 1 < n < m, if and only if C is n-connected in G(m,p) and 
C is not a proper subset of a subset of M which is n-connected. 
Definition 6: A subset C of M having three or more elements is a 
clique if and only if, for every i and 7 contained in C, i # 7, there 
exists an antimetry in G(m,p) from i to j; and, in addition, C is not 
a proper subset of a subset of M for which this is true. 


If G is the matrix representation of G(m,p) and A(n) = > G*, 


q=1 
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then the psuedo-structure B(n) on M has the matrix representation 
B= [bis]: 
bi; =1 if ai; > 0 and 17a; 
bi; =0 if ai; —0; 
b::=0 forall. 
The following statements are immediate consequences of the 
definitions 4 through 6: 


1. If U; are n;-connected subsets of M such that each pair U; 
and U; have a non-empty intersection, then the union of the U;’s is 
q-connected for some g,2<q<m. 


2. If U; are a sequence of n;-connected subsets such that each 
intersects the following, then the union of the U;’s is g-connected, 
for some g,2<q<m. 


3. If C is an v-connected subset (or an n-clique), then C is a 
subset of a g-connected subset (or a q-clique) for allq > n. 


4. Aset C isa cliqueif and only if it is a 1-clique. 


5. If G(m,p) is n-connected, then M is an n-clique. 
Theorem 4: A set C is an n-clique in G(m,p) if and only if C is a 
clique in the psuedo-structure B(n). 


Proof: 1. If C is a clique of Bin), then by Definition 6, bi; = 1, 
which implies a;; > 0 for i and 7, 1 #7, in the set C. Since, by 
definition of a clique, C has three or more members, it is an n-clique 
or a subset of one by Definitions 4 and 5. Suppose it is a subset of 
one, then there exists a k not in C such that ai, > 0 and a: > 0 for 
alli in C. This implies, by the definition of B(7), that bi, = bk: = 1, 
so C is a proper subset of a clique. Then, by Definition 6, C is not a 
clique with contradicts the assumption; hence C is an n-clique. 


2. If C is an n-clique, then a;; > 0 for all i and j contained in 
C with i #7. Thus, by definition of B(n), b;; = 1.Since C has three 
or more members because it is an n-clique, it follows that C is a clique 
in B(n) oris a subset of one. Suppose it is a subset of one, then there 
exists a &k not in C such that bi, = by; = 1 for all elements 7 in C. 
Hence, ai, > 0 and a;,; > 0 for all 7 in C, so the set C + {k} is n-con- 
nected. Therefore, C is not an n-clique, which is contrary to hypoth- 
esis; hence C is a clique of B(n). 


For convenience, Theorem 3 of (3) is restated: “Let S be the 
matrix of symmetries extracted from the matrix G representing a 
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structure G(m,p). An element 7 is contained in a clique of G(m,p) 
if and only if the i‘” entry of the main diagonal of S* is positive.” 


Theorem 5: If C and C’ are m-cliques of a structure G(m,p) and if 
there exists a q such that for some a contained in C and b contained 
in C’, ga“? = 1, then for any positive integer n, any 7 contained in 
C and any j contained in C’, gi; =0. 


Proof: Suppose there exists an integer n, an element 7 in C, anda 
7 in C’ such that g;;°"" = 1. Then for any c in C and din C’, 
Dea Pav Poa” = 1 and ga™gi;™ gj = 1, so C + C’ is m-con- 
nected, since if p + q + r > m and/or s + n + t > m the chains 
are redundant and there exist m-chains or shorter from c to d and 
from d toc. Thus C and C’ are not m-cliques for each is a proper 
subset of an m-connected subset. This contradiction implies that 
gij =0. 


Corollary: As sets, two m-cliques are disjoint. 


Proof: If i were a common element of C and C’, then for any 7 inC, 
gi; =1 and g;;‘" = 1 for some q and r < m, since i and 7 are both 
in C. But 7 is in C’, so the above theorem is contradicted. 
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ON THE EFFECT OF SELECTION PERFORMED ON SOME 
COORDINATES OF A MULTI-DIMENSIONAL 
POPULATION* 
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Samples are often obtained under circumstances which make it 
less likely to draw individuals from some parts of a population than 
from others. In this paper a method is presented which makes it 
possible, under certain assumptions, to correct for the resulting 
bias and thus reconstruct the means, standard deviations, and cor- 
relation coefficients of the original population. 


1. Introduction 


In obtaining samples from a multi-dimensional universe, one 
often is limited to a part of the universe only, or finds it easier to 
obtain values from some part of the universe than values from the 
remaining part. If, for example, a number of psychological traits is 
being studied, it is often not possible to test individuals drawn at 
random from a universe, since one may have access only to individ- 
uals who have passed certain admission tests or have undergone a 
screening process. This situation occurs in schools, armed forces, in- 
stitutional populations, etc. If it is desired to estimate the means, 
variances, and correlation coefficients of the universe, using such a 
sample, one clearly has to make corrections for the biased manner 
in which the sample was obtained. 

It is the aim of this paper to present a general technique for 
solving problems of this kind. While the formal mathematical theo- 
rem referred to is not new, its application for solving the problem 
of estimating population parameters from biased samples seems not 
to be generally knownj and deserves a detailed presentation. 

This type of problem was known at least as early as 1901, at 
which time Karl Pearson (6) formulated it and gave a method for 
solving it in the special case in which the original universe was 
multi-normal and in which a selection was performed on some of the 
coordinates in such a manner that the resulting population was again 


*Research done under the sponsorship of the Office of Naval Research. 
+See Campbell (2) and the discussion in Cochran (3). 
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multi-normal. Pearson’s method involved the laborious solution of 
systems of simultaneous linear equations. Taking up Pearson’s prob- 
lem, Aitken (1) set up the formal theory, again for the multinor- 
mal case, in the language of matrix algebra. A paper by Lawley fol- 
lowed (5), showing that the condition of normality can be relaxed. 
It is Lawley’s theorem which will be used as the mathematical start- 
ing point of this presentation. 


2. Notations. Formulation of the problem 


Before the general problem of this paper can be formulated ade- 
quately, it appears necessary to introduce the following notations 


and definitions. 
We consider the n-dimensional continuous random variable 


U= (X.,X2,°°+, Xp; Yorn» Vpn y***s Xn) 


and denote by 
x= (X,, Xe2,°++, Xp) 


the p-dimensional random variable of those coordinates of U which 
will undergo selection, and by 


Y= (Y Yo.2 br) oe Y,) 


the (n—p)-dimensional random variable of the remaining coordi- 


nates of U. 
Let the joint probability density of all coordinates of U in the 


parent population a be 
f(U) =f(X,Y) =f (X1, Xo 5++-, Xp, Vou ,-++, Yn), 
the marginal probability density of X in 2 
g(X) =g(X1,X2,-++, Xp), 
and the conditional probability density in 2 of Y for a given X 
h(Y/X) =h (Vpn » Ypre y*°* Yn/X1,X05°*+) Xp), 


so that we have 


f(U) =f(X, Y) =9(X) -h(Y/X). (2.1) 
The variance-covariance matrix of U in a will be denoted by 
V Tenn 
V — asl] oz P,P ’ p,n-p | : 99 
oul =] yer? yen’, | (2.2) 
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this way of writing indicates a decomposition of V into the variance- 
covariance matrix V,, of X , the rectangular matrix V»,»» of covari- 
ances between every X; and Y;, its transposed Vip» = V'pn», and 
the variance-covariance matrix Vn-pn-p of Y. 

The means (mathematical expectations) of the coordinates of U 
in a will be denoted by the column vector 














E(X,) 
E(X.) 
M, B(X,) 
eh d= || ake || ee ; (2.3) 
My E'CY on) 
E(Y,) 








A selection is defined as the modification of the probability den- 
sity g(X) into some other probability density g*(X). If the parent 
population a has the probability density (2.1) then the selected popu- 
lation a*, obtained when g(X) is changed into g*(X), has the prob- 
ability density 


f° (X,Y) =g*(X) -h(¥/X). (2.2) 


Under this definition, a mere restriction in range (truncation) 
of one or more of the coordinates X; is a special type of selection. 
Even a very general truncation consisting in the complete exclusion 
of some given region R(X,,---, X,) =F in the X-space is effected 
by replacing g(X) by 

GO ,ZeR 
cg(X) , Xekt 
and thus is a selection in the sense of our definition. Frequently oc- 
curring examples of such general truncation consist in excluding all 


values of U for which X, + X,+-+-+ X,<T, or for which X,<T,, 
As +A. < Fs, aa t+ + Ay ST, , GS 


We may now state our problem in the following form: 


9° (X) = 


+R denotes the complement of R in the X-space. 
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Problem. Let the means, variances, and correlation coefficients 
of population a* be known (for example by estimates obtained from 
a large sample). Under what conditions is it possible to reconstruct 
the means, variances, and correlation coefficients of the original 
population x? 


3. Lawley’s Theorem 


Given population 2 with probability-density f(X , Y) such that 
the following conditions are satisfied: 


(a) Y has linear regression on X , that is 
. p 
E(Y,/X) ={ es | Y; h(Y/X)dY pa +++ d¥n=djo + 3b: Xi, 
4=1 


(b) The conditional variances and covariances, for given X , of 
Yoni, Yo2,++:, Yn are independent of the value of X , that is, 


a (Y; : Y;/X) = | oe f 
[Y:—E(Yi/X)] (VY; —E(V;/X) ]h(V/X)dYp.. +++ dYn=Ci;. 


Then the following relationships hold between the variance-covari- 
ance matrix and the vector of means of the parent population 2 and 
those of the selected population 2°: 

















V pica || Vo» ’ Wandin ! 
WV Feieon 
jj ° @-PP » n-p,n-p || 
(3.1) 
Vow | Veo vw V "“—~ 
| Y tas or Fins F eaen ood Y wae { y*-,, a V""1.p Vp Vp} V"pn-p 
The further relationship 
M, = M*, = Y wae io (M, ame M’*,) (3.2) 


can easily be derived. 


4. Discussion of different cases of the problem 


We shall assume from now on that the conditions of Lawley’s 
theorem are fulfilled. From (3.1) and (3.2) it is obvious that knowl- 
edge of the means M* and the variance-covariance matrix V* of the 
selected population 2* alone is not sufficient to determine the means, 
variances, and covariances of the parent population 2. By adding 
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various additional information, we obtain different special cases of 
the problem formulated in Section 2. 

Problem 1. Let, in addition to M* and V*, the means M, and the 
matrix V,,, of variances and covariances of the coordinates on which 
the selection was performed be known. 

In this case (3.1) and (3.2) provide a complete solution of our 
problem, in a compact matrix notation. The remarkable feature of 
this solution is that no knowledge is required of the nature of the 
selection, since no such knowledge is needed for using (3.1) and 
(3.2). The selection may have been a generalized truncation with 
any region of exclusion R, or a simple restriction of range in one or 
several X-coordinates, or a selection which does not completely ex- 
clude any region in the X-space—formulas (3.1) and (3.2) will yield 
the same result, as long as M*, V*, M, and V,,, are the same, in each 
case. 

In particular, these formulas yield the solution of Problem 1, if 
ais multi-normal and the selection is such that 2* is multi-normal too. 
A detailed study of this case is given in a paper by Thurstone (8), 
where the derivations are based on Aitken’s theorem (1) and hence 
2 as well as 2* must be assumed multi-normal; the relationship be- 
tween the means in 2° and those in za is not discussed. 

The situation described in Problem 1 is of consi¢2rable practical 
importance. It arises, for example, if X,,---, X, are scores in stand- 
ardized admission tests (to college, to the armed forces, to a spe- 
cialized occupation), while Y,.,, +--+ , Yn are the outcomes of tests 
which can be applied only to those individuals who have been ad- 
mitted, e.g., achievement scores in the selected population. For such 
admission tests the parameters contained in V,,, and M, are usually 
known, while V* and M* can be estimated from a large sample. If 
now, for example, one wishes to know how the distribution of 
Ypa,-+t+, Y, will change if the admission standards in X,, --- , X, 
are raised or lowered, it is quite essential to reconstruct the whole 
variance-covariance matrix V of 2 and this is exactly our Problem 1. 


Problem 2. In addition to M* and V’*, let Vp.n-» = V'n-p» be given. 
This information is sufficient to determine V,_».n-» uniquely, since 
from (3.1) we obtain 


Vane Ee , chew + Y woe Fs TY san — V" an ol e (4.1) 
To reconstruct V,,, we must, however, solve the equation 


ona ‘gm « 
T aske — Fuse V ; 1 op Y ok . (4.2) 
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+1 
which is essentially a linear system of p(n—p) equations in 2+) 


unknowns, because of the symmetry of V,,,. If we know that 2* ac- 
tually is a population selected from z, a solution for V,,» must exist. 
Using this fact we see that, given a particular V*, the choice of Vpn-» 
is not altogether arbitrary; it must be such that the above equation 
has a solution for V,,,, i.e., the matrix of coefficients of the system 
must have the same rank as the augmented matrix. 

In the special case of Problem 2 in which p = 1, we have n—1 
equations in one unknown. This implies that the elements of Vp.n- 
are proportional to the corresponding elements of V*",,, V°pn—+. 

In another instance if p = 4(2—1) the solution of (4.2) re- 
duces to the solution of the linear system of p(n—p) equations with 
p(n—p) unknowns. 

In each of these cases it may happen that upon solving the equa- 
tions one obtains a matrix which cannot be part of a variance-co- 
variance matrix because it does not satisfy the necessary conditions 
of being symmetric and semi-definite positive. In such cases the as- 
sumption that 2* has been obtained by selection on (X,,---, Xp) =X 
from some population 2 whose variance-covariance matrix V con- 
tains the given component, and that the premises of Lawley’s theorem 
are fulfilled, must be false. 

Problem 3. In addition to M* and V*, we have Vn-pn-» given, 
which, by assumption, is a symmetrical matrix. To obtain the ele- 
ments of V,, we must solve a linear system of 4(n—p) (n—p+1) 
equations in 4p(p+1) unknowns. If a solution exists we then are 
confronted with the situation of Problem 1. 


5. Variances of estimates 


In many practical instances in which Problem 1 is solved, it may 
be assumed that M, and V,,, are known, while for the elements of the 
column vector M* and the variance-covariance matrix V* only esti- 
mates are available. These estimates are usually obtained by form- 
ing the corresponding means of a large sample from 2*. By substi- 
tuting these estimates, together with M, and V,,,, in (3.1) and (3.2) 
one obtains values for My, Vn-p,» and Vn-pn-» Which again are only es- 
timates. Their sampling errors are determined by those of the esti- 
mates in M* and V*. It would be clearly of interest to evaluate the 
variances of these estimates of the parameters in a. 

A method of solving this problem will be illustrated here only 
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for one estimate arising in the simplest 2-dimensional case. The 
method employed can obviously be extended to more involved cases, 
but the complications in algebra soon become prohibitive. 

Let a have the coordinates (X , Y) and let the selection be per- 
formed on X. We will derive an approximation to the variance of 


m,, the estimate for the mean of Y in the parent population z. 

Denoting by E* the mathematical expectation with respect to 
the distribution after selection, we see from (3.2) that in this spe- 
cial case 





m, = E(Y) = E*(¥) +) Ba) —EO)| ; 


Estimating the starred quantities by mean values we obtain 
aa8- 3a ae 
* b eg N N WN | =X 
v + Ms: — , 


My = 

















> xX? ( zy N 
N N 
=Y = pes S2zFT 
Letting t, = ,t =—_ , jf = —_, 4 = , we find 
N N N 


My = (ti, te, ts, be) 
ment t,(t— ts Mz) — t,4(ts — Me) 
t, — t,? ° 
We write t;° = E*(t;);7=1,2,3, 4 and 





= o(t, » te, ts, ts), 

¢° = p (t,°, t,°, t;°, t,°) . 
We now expand the function ¢ into a Taylor series about the point ¢°, 
‘2 
g= et 32 (te) +. 


According to a well-known theorem of Doob (4), ¢ — ¢° is asympto- 
tically normally distributed with expectation 0 and variance 


id oe. 1] *| 
E}[ 3% tt ney} } , 


so that we have 
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o (my) =E'| 3 “= 


i=1 yj 


2 
. 





(1-49) | 


Let d = m,* — m;; we finally have the approximate expression 





A Ba | 2 2 
N ot(my) = [1+ 2)" | oper) +0 E (oe | 




















(o,")? Cr 
p?.0,? d2 (o,,°)? p? 0,2 ; 
- + . | (ost) + dm, my + met) | 
a7 (o,")* 077 (0;")? ig 
dc * 
ad? 2 le Nye / P oy y,22 
+ (o(2y)*) 4 2| 1+ m y 
(o2")* (oz")? oz(o2")? 
2 i+ me pv *j2igd * * 
= (c2")* igo | ( ) . ayy si Me ) "Dy 
“d do 4 
Mz YT 
of. 4] 
(oz as (oz )? 
—? 2 2 
thes (o,°)? + d( mz — My + m,*) “a 
ose, )* 22,2 
—2po,do,.* 
dl a be | *)? + d(m.—™. +m.) 5 
Or ' Niby L C . 
az(oz")* a a;(o,")? 4 v,ry 


6. Numerical illustrations 


Example I. A sample of 942 individuals was obtained from a 
population of test scores with the five coordinates (A,B,C,D,F). 
The original tests were so constructed that (A, B, C, D, F) had 
approximately a five-dimensional non-singular normal distribution. 
The 942 individuals were obtained at random, with the following 
rather complicated selection. Only individuals scoring at least 40 in 
test F were to be included, which should have resulted in an out- 
right truncation F > 40; however, candidates whose score F would 
have been slightly below 40 were helped and up-graded a bit, hence 
scored 40 or more, and were included in the selection. 

We now consider the population 2 of the 942 selected individ- 
uals. Since in the original population (B, C, D) had linear regres- 
sion on A and F and the conditional variances and covariances of 
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(B,C, D) were independent of A and F, these two conditions still 
hold (approximately) in 2; and Lawley’s theorem is applicable if a 
further selection is performed on the coordinates (A, F’). 

The scores for the 942 individuals of x were available on IBM 
cards. The actual values of the correlation coefficients, the standard 
deviations and the mean values in 2 were computed and are given in 
Table 1. 











TABLE 1 
A B Cc D F 

A ul 0.703 0.527 0.499 0.777 

B i 0.404 0.555 0.679 

Cc ; a 0.253 0.578 

D 1 0.427 

F 1 
Mean 46.90 47.56 48.61 50.18 50.25 

o 9.93 9.39 8.80 7.19 8.05 





A selected population 2* was obtained by imposing on (A, F) 
the truncation A >—o,F > 45. After this truncation there were 
647 individuals left in a*. The first and second moments in z* were 
computed; they are 


F A B C D 














F || 48.00 37.65 37.24 25.59 14.46 
A || 37.65 72.24 48.95 29.04 23.90 
v= (6.11) 
B || 37.24 48.95 82.60 20.10 31.24 || , 
C || 25.59 29.04 20.10 68.04 4.71 
D || 14.46 23.90 31.24 4.71 48.31 
54.2195 | F 
50.9706 A 
M* = : (6.12) 
50.4544 B 
51.2117 C 
51.8439 D 














The first and second moments for A and F, computed exactly 
from the 942 cards in 2, were assumed as known. They are 


F A 
F || 64.80 62.11 
Vip = (6.21) 
A || 62.11 98.60 | oat 
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Fi) 50.2500 | 
M.= | | - (6.22) 
A | 46.9000 | 
Using the values (6.11), (6.12), (6.21), (6.22) in (3.1) and (3.2), 
we obtain the missing first and second moments in 2, 


_ || 5848 39.55 23.39 || 
aul | 72.38 44.54 33.63 || ’ _— 
| 103.26 33.72 39.87 | 
Vaowe= || 33.72 77.01 10.41 || , (6.32) 
|| 39.87 10.41 51.90 
| 46.76 | 
M,= || 48.74 || . (6.33) 
50.33 | 








Converting the covariances and variances in (6.21), (6.31) and 
(6.32) into correlation coefficients, we finally have the following “re- 
construction” of the parameters in the parent population 2: 














TABLE 2 
A B C D F 
A 1 0.717 0.511 0.470 
B 1 0.378 0.545 0.715 
C 1 0.164 0.560 
D 1 0.404 
F 1 
Mean 46.90| 46.76 48.74 50.33 50.25 
. 9.93] 10.16 8.78 7.20 8.05 




















The boxed values were assumed known. 


A comparison of Table 2 with the “true” values in Table 1 
shows apparently good agreement. How good it really is could only 
be judged if at least approximate values of the standard deviations 
of the values'in Table 2 were computed. This has not been done. 


Example II. A three-dimensional normal parent population 2 
with the coordinates (X,, X., X;) was constructed in the following 


manner: 


The parametets were given the values 
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E(X:i)=m,=0 for 1=1,2,3 
o7 (X;) =e;??=—1 for +=1,2,3 
=p, = 08, 


(6.41) 


P22 


so? 


ep, =O. 


Pots 72,23 


The frequencies in the cells of Tables 3 and 4 were computed ac- 
cording to the probabilities obtained from Table VIII, Part 2 and 
Table II, Part 1, respectively, in Pearson (7). 


A population 2* was defined by the following truncation 
X,+ X; <—0.4, X, >—0.8. (6.51) 
In Figure 1 the shaded area indicates the portion of the (X2,X3) 


X; 


8 





~A 


-B 
4 


FIGURE 1 











plane in which all the (X.,X;) coordinates of the individuals con- 
tained in 2* must lie. 

A sample of N = 93 individuals was drawn from a* with the aid 
of a table of random numbers. The following mean values were com- 
puted from this sample 





>= XiX2 ae 

= 0.21376 DT =—0).09247 
bo XiX3 = X;? 

= —0.11699 = 0.25355 








N 
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D XX TX 
“= —0.11097 “= —0.79855 
< 2 2 Ae = 1.8567 
: = —0.07527 a : ff 
a 2 


0.47699 
“ad . 7 e 


From these values, all the parameters of x* were estimated: 


m,* = —0.07527 o,2%== 0.47182 
M2" = —0.09247 o"2= 0.24500 
m,* = —0.79355 o32= 1.22705 
10"= 0.20680 G1s* = —0.17672 


wan = —0.18435 . 


Hence we obtain 








Xe aa me 
X, || 0.24500 0.18435 0.20680 | 
x, | —0.18435 1.22705  —0.17672 || 
v= . 
X, || 0.20680 —0.17672 0.47132 | 
X, |) —0.09247 | 
M*=X, || —0.79355 | . 
x, || —0.07527 | 








Since the selection was performed on (X,, X;), We assume as 
known a priori the parameters 


M,=™M3=0, o2=o3—1, ps0, 
and solve the problem of the first kind. The results obtained are 
m, =—0.014 o = 0.992 
pi2— 0.836 pis = —0.020 
and appear reasonably close to the parameter values 


m =—0, o.—1, pi2 — 80, pis — 0. 
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PSYCHOMETRIKA—VOL. 15, NO. 2 
JUNE, 1950 


THE RELATION OF THE PSYCHOMETRIC SOCIETY TO 
THE FEDERATION OF STATISTICAL SOCIETIES 


Since the recommendations made by the Psychometric Society’s Committee 
on the Relation of the Psychometric Society to the Federation of Statistical 
Societies will be considered at the September 1950 meeting of the Psychometric 
Society at State College, Pennsylvania, an analysis of the problem made by the 
Chairman of the Committee is presented here. It is followed by the final report 
of the Committee. 


1. THE PROBLEM OF RELATIONSHIP BETWEEN THE PSYCHOMETRIC 
SOCIETY AND THE AMERICAN STATISTICAL ASSOCIATION 


Two types of relationship are indicated in the ASA Constitution, association and 
affiliation. 


Definitions 





Associated Society 

An associated society is one inter- 
ested in the objects of the Associa- 
tion and concerned with the advance- 
ment of statistical methods or with 
the statistical aspects of a content 
field. 

Associated 

Designates two members to serve on 
ASA Council. 

Designates one member from the 
Editorial Board of each of its pub- 
lications to serve on the Commit- 
tee on Publications. 

May name a representative on the 
Editorial Board of the News Bul- 
letin. 


Affiliated Society 
An affiliated society is one whose 
objects are related to those of the 

ASA in such a way as to make co- 

operation advantageous to both 

groups. 
Affiliated 

Designates one member to serve on 
ASA Council. 

An affiliated society is not repre- 
sented on the Committee on Pub- 
lications or on the Editorial Board 
of the News Bulletin. 


Provisions Applying to Both Statuses 


1) Publications of both organizations become available to both groups on the 


same basis at lower rates. 


a) This means $5.00 for the Journal of the American Statistical Asso- 
ciation and $1.00 for the American Statistician. Would members of the 
Psychometric Society prefer to join the ASA for $8.00; subscribe to the 
Journal of the American Statistical Association for $5.00 or to the 
American Statistician for $1.00; or neither? 
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b) Members of ASA may subscribe to Psychometrika for $4.50. Would 
we lose members who like to save $0.50? How many of the three or 
four thousand members of the ASA would be interested in the privilege 
of subscribing to Psychometrika at $4.50 instead of the regular sub- 
scription price of $10.00 but would not be interested in joining the So- 
ciety at $5.00? 


2) At least 3 months notice must be given before the relationship can be termi- 
nated. The relationship may be terminated by the affiliated or associated 
society or by a 2/3 vote of the Council of the ASA. 


8) Members of the Psychometric Society become associate members of the ASA 
—members living outside the United States and Canada may be excluded 
from the privileges and responsibilities of membership. 


(Is this associate membership free of charge or are there associate mem- 
bership dues? The back cover of the American Statistician does not men- 
tion associate membership. The By-Laws, Article I, Paragraph 9, Associ- 
ate Members, says: “Each Associate Member shall receive the news bul- 
letin and have their [sic] name printed in the Directory of Members.” It 
looks as if we might be increasing the dues of the Psychometric Society 
by this procedure in that we would be forcing the members of the Psy- 
chometric Society to pay associate membership dues to the ASA. The 
Committee has been informed that the ASA Board of Directors will take 
up this question for further clarification. This Board has already voted 
“no dues” for “Associated” Members. Mr. Weiss, Executive Director of 
the ASA, is in favor of optional subscriptions to the news bulletin at 
$1.00.) 


4) The Psychometric Society will not be limited in any way by the ASA in elect- 
ing its own officers, deciding membership qualifications, editing its own pub- 
lications, etc. 


(Advantages might accrue to all groups concerned with the general prob- 
lem of measurement by affiliation with a basic statistical organization. It 
is intended that some clearance arrangement will be made through the 
Committee on Publications for appropriate channeling of papers to re- 
spective editors, stimulating publications in various areas, and on a co- 
operative basis to cover the entire field of interest of the respective 
groups, thereby increasing knowledge in various subject-matter fields.) 


5) By-Laws, Article II, Section 7: Financial Relationship with Associated and 
Affiliated Societies. For all services provided to associated societies, includ- 
ing annual billing and collection of dues, the production and distribution of 
joint directories, and employment and placement services, joint costs will te 
distributed to the Association and the associated societies in direct propor- 
tion to the number of members included on each society active membership 
list. All income deriving from jointly distributed publications will be divided 
between the participating societies in direct proportion to their total paid 
circulations. All other joint activities will have their expenses or incomes 
divided in proportion to membership or publication circulation. All specific 
services provided associated societies or chapters, not participated in by other 
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societies, will be charged at cost or in any other way approved by the Board 
of Directors. 


(The Committee is informed that this section was meant to apply indif- 
ferently to both associated and affiliated societies.) 


2. FINAL REPORT OF THE COMMITTEE ON THE RELATION OF THE 
PSYCHOMETRIC SOCIETY TO THE FEDERATION OF STATISTICAL 
SOCIETIES 


The Committee recommends that the Psychometric Society apply for the status 
of “Associated” society in the American Statistical Association. This request 
should be approved by the Psychometric Society (or disapproved) by some appro- 
priate procedure and should include 


1) a copy of the Constitution of the Psychometric Society, 
2) a list of members, 

38) a list of current officers, and 

4) a financial statement. 


The status of “Affiliated” society seems to the Committee members less de- 
sirable than the status of “Associated’”’ society. 

The Committee has not considered in detail point 5 of the foregoing memo- 
randum on the problem. It is the Chairman’s opinion that the matter of finan- 
cial relationship of the Psychometric Society with the American Statistical Asso- 
ciation should be studied more carefully. It may be that our present system :s 
so efficient that it would be unwise to have billing, collection of dues, etc., han- 
dled by the ASA office. 

HAROLD GULLIKSEN, Chairman 
Harry H. HARMAN 
JACK W. DUNLAP 


COUNCIL OF DIRECTORS NOMINEES 


The Council of Directors has nominated Lee J. Cronbach, 
Paul Horst, E. F. Lindquist, and Charles I. Mosier for the 
two three-year terms on the Council of Directors beginning 
October 1, 1950. Two of these nominees will be elected at the 
Annual Meeting of the Psychometric Society at Pennsylvania 
State College next September. 
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Candidates Still Wanted for 
CHIEF, TEST DEVELOPMENT SECTION IN THE 
U. S. CIVIL SERVICE COMMISSION 


The United States Civil Service Commission has announced that 
it is still looking for well qualified candidates for the position of Chief 
of its Test Development Section at a salary of $10,000 a year. A new 
Notice of Special Recruitment has been issued setting forth the duties 
and special requirements for this position and extending the closing 
date from April 27, 1950, to July 5, 1950. 


The person selected to fill this position must have outstanding 
qualifications. He will direct a staff of professional workers in a 
broad-gauge program of research and development in the field of per- 
sonnel selection, as well as supervise day-to-day production opera- 
tions. This will include work on the development, improvement and 
production of a great many kinds of tests for large-scale examining 
programs. 

Applicants must show that they have had successful adminis- 
trative experience in planning, directing, and coordinating the devel- 
opment, construction, and validation of tests and other personnel se- 
lection devices for a broad range of occupational areas and must have 
earned recognition as leaders in the field of personnel measurement 
and evaluation. 


Full information regarding the duties and the requirements for 
Chief, Test Development Section, and instructions on how to apply 
are given in the Notice of Special Recruitment issued for this posi- 
tion and in Examining Circular No. 9 for Research Psychologist. 
Copies of these forms may be secured at the Commission’s regional 
offices or by writing direct to the U. S. Civil Service Commission, 
Washington 25, D. C. Copies may also be consulted at most first- and 
second-class post offices throughout the country. 


Applications should be filed under Examining Circular No. 9 and 
must be received in the Commission’s central office not later than July 


5, 1950. 
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BOOK REVIEWS 


QUINN McNEMAR. Psychological Statistics, New York: John Wiley and Sons, 
1949. Pp. vi + 364. $4.00. 


Compared with other texts in the field of statistics, which attempt to cover 
the same material at a relatively simple level, this book is refreshing. It repre- 
sents an honest effort to convey a true, workable understanding of statistics 
rather than simply providing a number of computational methods which can 
be applied. While teaching statistics to students of psychology will continue, I 
fear, to be a difficult problem, a text such as this, which stresses understand- 
ing of statistical techniques, their meaning and limitations, and the theory be- 
hind them, should help in getting across much that will be valuable to the 
student. 

As one would find in any book on statistics written at this level, and writ- 
ten by one person, there are inadequacies here and there. Most people, I sus- 
pect, will find the last chapter on sampling and statistical inference incom- 
plete, inexact, and relatively useless. The book might have been a better book 
without this last chapter which, unfortunately, introduces several confusions. 

Other inadequacies are mostly of the character of omissions in explanation. 
I shall select a few examples of these to illustrate the type of thing involved: 

(1) On page 64, in discussing the sampling distribution of differences be- 
tween means, the author states, “It will be noticed that we can have a corre- 
lational term in formula (26) only when it is possible to pair, on some basis 
other than chance, the scores which enter into the first mean with the scores 
which contribute to the second mean.” The formula involved is the standard 
deviation of a distribution of differences between means. The author manages 
successfully to convey the impression that the correlation term exists theo- 
retically only if one can compute it in practice. This is, of course, not true. 
There are many instances where correlation can exist between the means, but 
this correlation cannot be estimated from the sample. The student should know 
that, even though he cannot estimate the correlation and assumes it to be 
zero, he may be in error in so doing. 

(2) On page 98, in summarizing his conclusions after a discussion of 
rank order correlation, the author states “Rho is not as consistent from sample 
to sample as r, does not possess the mathematical advantages inherent in 7, and 
therefore has merit only when the observations involve ranks, i.e., are not meas- 
ures.” It is somewhat surprising that the author, who is clearly sensitive to 
problems of statistical inference and testing, and who is concerned with limi- 
tations and inadequacies involved in making the necessary assumptions about 
the tests employed, can ignore the important role of rho in being amenable to 
exact tests of significance which do not involve assumptions concerning the par- 
ent population. This use of rank order correlation has been developed and used. 
It is too bad for a book of this caliber to dismiss an excellent technique so sum- 


marily. 
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(3) On page 166, discussing limitations of the product moment correla- 
tion, the author says, “The product moment correlation ..... needs careful 
qualifying if either or both variables yield skewed distributions.” And in the 
next paragraph he says, “There are no general rules to follow in the case of 
variables yielding skewed distributions. Frequently, one can use a logarithmic 
transformation of such a variable and thereby secure scores which are at least 
approximately normal; or one may deliberately normalize the distribution by 
converting the raw scores into T scores.” It is never made clear by the author 
just how or why the product moment correlation needs careful qualifying if 
we have skewed distributions, and it is certainly not made clear how we are 
helped by such a dubious procedure as normalizing the distributions involved. 
It is common knowledge that the calculation of a product moment correlation 
involves no assumptions whatsoever concerning the normality of the variables 
involved and that interpretation of the correlation coefficient can be made un- 
equivocally in terms of a best fitting straight line irrespective of the distribu- 
tion functions. It is true, of course, that to the extent that we do not have a 
normal correlation surface, we are at somewhat of a loss in testing significance 
of correlations. This latter point, however, is certainly not helped by delib- 
erately normalizing the distribution. It is not clear what the author has in 
mind or why he makes the recommendation. 

It should be stressed, however, that such points as these might not even 
be points of criticism were the book otherwise not on such a good level of ex- 
planation. I would like to devote the rest of this review to a discussion of two 
basic questions which this book brings up about the relationship between sta- 
tistics and psychology. 

The basic place of statistics in relation to any science or any body of data 
is as a tool or as a servant. Statistics does not dictate to the science, but rather 
seeks to adapt itself and to develop so as to help the science. In psychology 
today we are in danger of losing this nice relationship with statistics. Sta- 
tistics has in many ways been oversold to psychologists and is rapidly becom- 
ing a tyrant, inflexible and nonadaptive. There are two major aspects of this 
trend toward statistical tyranny, both of which show themselves well in Mc- 
Nemar’s book. One of these concerns the question of what is and what is not 
significant statistically, and the other concerns the question of the value placed 
upon high-powered statistical techniques in themselves, rather than in relation 
to the data on which they are used. We shall discuss these two points in order. 

(1) The level of significance to insist upon: 

Fifteen years ago statistics textbooks that were used in connection with 
psychology tended to make the simple and rather dogmatic statement that a 
critical ratio of 3.0 was needed in order for a difference to be regarded as sig- 
nificant. Later when R. A. Fisher made his considerable impact upon statistics 
and statistical thinking, this was modified in most textbooks to permit the in- 
vestigator to also consider P values of .05 and .01 as evidence for regarding a 
difference as significant. 

It has always been said that the particular P value upon which an inves- 
tigator insists is an arbitrary matter. This, however, has not prevented dogma 
in relation to the selection of a P value. When the notion was introduced into 
statistics of two types of errors, namely, accepting the null hypothesis when 
false and rejecting the null hypothesis when true, the reasoning behind the 
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selection of what P value to insist upon became more sophisticated but the rec- 
ommendations of writers still remained fairly dogmatic. In most cases the rea- 
soning went something like this: There are two types of possible errors and 
the investigator must choose his P level according to the relative importance 
he places upon these two kinds of errors, but nevertheless one should insist upon a 
high level of confidence for rejecting the null hypothesis. This type of sophis- 
ticated non sequitur is very well illustrated in McNemar’s book. After a very 
excellent discussion of the two types of errors, McNemar concludes on page 67: 


At the other extreme, a few are willing to accept as significant a 
difference which is 1.5 times its standard error. Since P = .13 for aCR 
of 1.5, it is readily seen that such persons would all too frequently 
have their publics believing that chance differences are real. A less 
lax level, which has had general acceptance by some workers, is repre- 
sented by a P of .05, or a CR of nearly 2.0. This is also a rather low 
level of significance for announcing something as “fact.’”’ Those writers 
who advocate the .05 level for research workers in psychology, soci- 
ology, and education cite R. A. Fisher, the world’s leading statistician, 
as their authority, but they fail to point out that Fisher’s applications 
are to experimental situations wherein there is far better control of 
sampling than is ordinarily the case in the social sciences. 





In short, after discussing the two types of errors, McNemar seems content to 
base his conclusions about P values on only one of those types of errors. He 
seems to regard only the rejection of the null hypothesis as “announcing some- 
thing as fact” and does not seem to feel that accepting the null hypothesis is 
also announcing something as fact. If there is something in between rejecting 
the null hypothesis and accepting the null hypothesis, McNemar fails to point 
out what this third alternative is. He also fails to support the somewhat spe- 
cious reasoning that a lower level of significance is appropriate to experimen- 
tal situations. 

McNemar also states on page 69, “If some reader must have a criterion re- 
gaiding what is or is not significant, the author suggests that he compromise 
by taking the level indicated by a P of .01.” Again in the midst of an excellent 
discussion concerning the kinds of errors of statistical inference that are in- 
volved, the conclusion is drawn that one must have a very high level of confi- 
dence for rejecting the null hypothesis. This, morefer, does not express all of 
McNemar’s opinions on the matter. On page 233,’ in discussing small sample 
methods, we find the following interesting statement: “It seems to us that those 
who publish statistical results based on a small number of cases should, unless 
they are positively sure that the basic assumptions underlying it have been met 
(and this assurance can seldom be attained), adopt a more stringent level of 
significance, says a P of .001 before drawing definite conclusions, and a P of 
-01 as the borderline of significance, i.e., as suggestive.” 


Let us examine whether such recommendations and such opinions are valid 
conclusions from the arguments which are and can be made, or whether these 
opinions and conclusions stem rather from prejudices. The tabulation below pre- 
sents an example of how frequently the error of accepting the null hypothesis, 
with a specified true difference existing, would occur with different numbers of 
cases in the samples and specified P values for rejecting the null hypothesis. 
The example presupposes a true difference of +5 and a standard deviation of 
differences equal to 10. 








212 PSYCHOMETRIKA 


Proportion of Time Null Hypothesis Would Be Accepted Where True 
Difference between Means is 5.0 and o = 10 


P Value Required for Rejecting 





Number of Null Hypothesis 
Cases o .003 01 05 13 
M1-M2 
9 33 95 84 .69 .50 
25 2.00 .69 50 81 .16 
49 1.43 24 .08 .02 .002 
100 1.00 05 01 -003 .0002 


It can be seen that for small numbers of cases, if a true difference exists, 
there would be a tremendous number of errors committed by insisting upon a 
high level of confidence for rejecting the null hypothesis. Thus, in this exam- 
ple, with an n of 25, if we insisted upon the 5% level of confidence, 31% of the 
time we would accept the null hypothesis erroneously. Only with a relatively 
large number of cases can one set a high level of confidence for rejecting the 
null hypothesis and also be reasonably sure that he will not be committing a 
large number of errors of the other type. 

If, then, in some instance both types of errors were equally important, our 
choice of level of significance for rejecting the null hypothesis would tend to be 
very different from what is customarily employed in psychology today, and radi- 
cally different from what is recommended by McNemar. In fact, we would come 
to one conclusion exactly contradictory to McNemar. He suggests that the small- 
er the number of cases the higher the level of confidence we should insist on 
before rejecting the null hypothesis. From the example in our tabulation we 
would, however, tend to conclude that this would greatly magnify the number 
of errors of the other type which we would make. It is probably more reason- 
able to insist upon less confidence in rejecting the null hypothesis when the num- 
ber of cases is small. The criteria for rejecting the null hypothesis can become 
stricter and stricter as our number of cases increase. In an a priori sense, one 
should not expect as much confidence from small samples as from large ones. 

It is important to realize that a conclusion of no difference is frequently as 
important as concluding that there is a difference. Both of these types of con- 
clusions can have equal implications for action. It is consequently important to 
choose a level of confidence which tends to minimize both kinds of errors rather 
than maximize one kind of error while minimizing another. Statistics must be 
looked at as a help in understanding and interpreting our data rather than as 
a determiner of specified standards of a statistical nature which should be met. 

(2) The self-perpetuating character of statistical techniques: 

It is unfortunate for psychology and for psychologists that very few of the 
commonly used statistical techniques have been developed in connection with 
the practical problems of analyzing psychological data. The result is that ex- 
cellent statistical techniques whose assumptions are geared to conditions of data 
in other fields are now used in psychology. The attempt is made to force the 
design of psychological experiments into the required pattern which would then 
make them amenable to treatment by these statistical techniques. The question 
is rarely raised of the efficiency of these designs and of these statistical tech- 
niques for the problems involved in psychology. McNemar, in spite of his ex- 
cellent treatment of the conventional statistical techniques, is a good example of 
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an author paying much attention to the requirements of the statistics and rela- 
tively little attention to the requirements of the data which the psychologist is 
interested in gathering. This omission is made even more glaring because the 
book is entitled Psychological Statistics, carrying the implication that it is for 
the psychologist. 

Thus, in the book we find little attention being given to such techniques as 
rank order correlation, tests of significance based on rank orders and serial or- 
ders, and other such devices which can be of great help in many kinds of psy- 
chological data. Nowhere in the book can one find an attempt to teach the stu- 
dent how to use elementary probability theory in connection with his data. 
There is no elaboration of such simple devices as, for example, six independent 
differences in the same direction being significant at the 3% level of confidence, 
or a predicted order of five quantities being significant at the 1% level in terms 
of the possible permutations that could have occurred. Knowledge and use of 
such simple devices, however, is more important in many fields of psychology 
than complicated techniques such as analysis of covariance, to which the author 
does devote considerable time. 

There is altogether no attempt to search the statistical literature for tech- 
niques and devices which are suitable for the peculiar conditions of psychological 
data. On the contrary, the book is merely an excellent presentation of statis- 
tical techniques which are best applicable to other fields than psychology. The 
discussions of the use of chi-square and of analysis of variance, for example, 
are the standard kinds of presentation for use where the experimental varia- 
tions do not form a continuum. So frequently in psychology, experimental 
groups do fall along a graded continuum and nowhere in McNemar’s book will 
one find the statement that chi-square and analysis of variance are inefficient 
techniques when the experimental groups are degrees of variation along a con- 
tinuum since these techniques ignore the ordering of the results. It will have 
to be from books other than this one that psychologists will be able to learn to 
use statistics as a flexible tool rather than submit to statistics as a rigid master. 


University of Michigan. LEON FESTINGER 


On Festinger’s Review of Psychological Statistics 

Naturally I applaud certain parts of Festinger’s review and I find sugges- 
tions therein that are worth filing for a possible revision of the book. But I also 
find points of fact and of philosophy with which I disagree. Of course I can say 
nothing to the nonparticularized criticism that the last chapter is “incomplete, 
inexact, and relatively useless,” and that it “introduces several confusions.” 

A word on Festinger’s third illustration of inadequacies. His implication 
that I think normality of distribution is assumed for r is controverted by my 
statement on p. 118, “7x does not assume normal distributions.” He also evi- 
dently missed my reasons, stated on the same page, for injecting a word of cau- 
tion when nonnormal distributions are involved. My suggestion that distribu- 
tions be normalized by transformation is said to be a “dubious procedure”—per- 
haps as a strong advocate of rank correlation the reviewer would prefer to 
rectangularize the distributions. He states that the testing of the significance 
of correlation “is certainly not helped by deliberately normalizing the distribu- 
tion.” Regarding this generalization, we refer the reader to the article and 
bibliography of Mueller (Psychol. Bull., 1949, 46, 198-223). 
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Readers will have noted that Festinger uses a large portion of the review 
to expound his own ideas on statistics in psychology. He seems to be worried 
about statistics being “oversold” to psychologists, “becoming a tyrant, inflexible 
and nonadaptive,” a “rigid master.” I am indicted for aiding and abetting this 
“trend toward statistical tyranny.” 

In partial support of the above apprehensions, one-half the review is de- 
voted to alleged dogma concerning what is significant statistically. Anyone who 
reads pages 66-69 and 232-234 may judge for himself whether the exposition 
is “dogmatic” and “prejudiced” or whether Festinger’s discussion, and his use 
of quotations out of context, misrepresent these pages. Furthermore, there are 
points in his argument which demand comment. 

When he says that “accepting the null hypothesis is also announcing some- 
thing as fact,” we ask what kind of fact emerges except the trivial fact that 
the null hypothesis couldn’t be rejected. When he says that it is “important 
to choose a level of confidence which tends to minimize both kinds of errors,” 
we ask whether there is a method for doing this. A search of the literature 
indicates that this problem has not been solved. (An outstanding mathematical 
statistician informs me that simultaneous minimization of the two types of er- 
rors is simply impossible.) 

When the reviewer supposedly shows by a cleverly devised example that 
my expressed lack of faith in small samples is ill-founded, he unwittingly pro- 
vides an excellent illustration of my claim that small samples are not condu- 
cive to rejection of the null hypothesis. Moreover, he does not, and for reasons 
which will soon be obvious, carry his own argument to its logical conclusion. 
Suppose that instead of a true difference of +5 it is presumed that the true 
difference is +1. Then with an N of 25 and the .05 level for significance, one 
would erroneously accept the null hypothesis 98% of the time. Now to over- 
come this sad state of affairs, let us follow the reviewer’s suggestion of using 
a less stringent level of significance. A little arithmetic shows that even to 
reduce the 98% to 50% would entail using a critical ratio of only .5, or a P 
level of about .62, for rejecting the null hypothesis! 

Being ever aware that statistical inference is a part of the logic of scien- 
tific method and being always wary of the use of any “flexible” logic, I regard 
Festinger’s concluding sentence as a compliment. Submitting to statistics as 
a “rigid master” involves nothing more than submitting to the rigors of scien- 
tific method. 


Stanford University QUINN McNEMAR 
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